We're in the same boat. You're not doing anything wrong. I read someplace (an Infiniti help page, I think) that iPhone phone books are not supported yet. I also have a blackberry work phone, and it won't download that phone's contacts, either. Lame. I think that's because there are too many, but I'm not sure. I manually entered contacts. By the way, I find the voice recognition to be much better with nav than it is with phone contacts.