Zero is black. The number one is yellow. Two is blue, a dark shade of blue. Three is also yellow. Four is purple, sort of a mauve or else dark violet. Five is pink, or bright orange. Six is yellow-orange. Seven is lime green, eight is deep indigo, and nine is vivid red.

According to Wikipedia, Synesthesia (or ideasthesia) is "a neurological condition in which stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway....It is estimated that synesthesia could possibly be as prevalent as 1 in 23 persons across its range of variants."
I found this information at the Science Centre in Toronto, as it so happens, about a year ago... no, longer than that. Anyway, I thought it was interesting.
I have synesthesia, although previously I hadn't known what it was called. I have grapheme-colour synesthesia, which is association of numbers or letters with colours. I also have motion-sound synesthesia, meaning I attribute sounds to every movement that I see or feel. I also associate personalities with numbers, for example, one is very famous, but also humble. Two is laid-back. Three is a workoholic. Four is quiet and withdrawn. Five is crazy, spontaneous and likes to party. Seven is tragically misunderstood. Eight is chubby and probably owns a restaurant or something; welcoming and kind, but also loud. Nine is bossy; a control freak.
I also hear the words spoken out loud in my head, when I read text; does everyone have that? Different fonts invoke different-sounding voices, and different expressions. I also associate places with colours, and languages with colours. In fact, I think maybe it's just the way the word sounds that makes it have a colour. Sometimes letters have colours as well. So really, it's not the images that invoke colours, it's the sounds that I associate with them. Same for numbers. Sometimes I also relate words to words that they rhyme with, so I might hear "mice" and think of coldness, and then realize that I was associating with "ice"... anyways, apparently synesthesia runs in families. Maybe some other people in my family have this as well. Or maybe I'm just crazy.