I'm not a New Yorker, but I have been there, and
Other Music (15 E. 4th Street) and
Downtown Music Gallery (342 Bowery, between E. 2nd & E. 3rd Streets) are great.
Excuse my vagueness, but there are also two or three great shops on Macdougal Street, Sullivan Street or Thompson Street in Greenwich Village, somewhere around West 3rd Street or West 4th Street (there's also a fantastic wholefood cafe on either West 3rd Street or West 4th Street), and there are also loads on St Marks Place. If I can find more helpful details at home I'll post them.
EDIT - One of the shops in Greenwich Village is Generation Records at 210 Thompson Street.