So far we have:
Metroid Prime: Hunters- Walkie talkie like chat only in lobby.
Custom Robo Arena- Walkie talkie like chat only in lobby.
Pokemon D/P - VoIP phone like communication before and after battle.
Planet Puzzle League - VoIP phone like communication before, after and during matches. This is my default DS Wifi phone game

.
Advance Wars: Days of Ruin - VoIP phone like communication before, after and during matches. Haven't played the game much yet though...
There's a homebrew called (guess what)
Voicechatclient that has a board for scribling and support for several people to chat at the same time. It works and it's rather nice. You might wanna check it out.