The question made me stop and think about the reasons I had started doing more and more black & white images.
I think that a big part of it is what my friend suggested: it allows your mind to fill in the color. But for me there are 3 additional reasons that I tend towards black & white:
Making it simple
To me, black & white is more pure. When I do a painting the main tool for composition is value (the black & white scale); the colors are just a complication. When I do a photograph I try to reduce all distractions. I do this by cropping (and thus coming in closer) when I take the image. I also try and use a shallow depth of field so that the background will be out of focus. When I process the image on the computer I might remove distracting objects: power lines, garbage, extra people, etc.
Developing the image in black & white is an additional way to simplify and remove distraction. If the color doesn't help tell the story of the image, I eliminate it.
Part of it is hype: we tend to associate b&w with fine art, what we would see in a museum. When I show an image in black & white it is no longer ‘just' a photograph, it is art 😉
Working Within My Limitations
A last consideration for me is that I am color blind, so I am a lot more comfortable manipulating tones than I am colors. I know that the tones will appear to the viewer as they do to me, whereas others will see colors very different from me.