There's a veggie place that I've recommended two or three times in "I'm going to New York" threads. It always takes me ages to find its name, and I haven't the time right now, but if you do search for the other threads, you'll find it somewhere. It's somewhere around the junction of 3rd or 4th Street and Sullivan, I think.
EDIT - Found it! It's Quantum Leap, and is at 88 W 3rd St, btwn Thompson and Sullivan.
Reviews here.
Record shops? Other Music, Downtown Music Gallery, Kims, several others on St Marks.