I did just that yesterday. In a hurry to get some content live for my website I accidentally pushed a config file to GitHub containing my Twitter, Tumblr and Moves API keys.
My first instinct was to immediately head over to GitHub and delete the file... which of course is simple since file deletion goes against everything a VCS like Git tries to achieve. Deleting the file would only remove it from that version while previous commits and history will still show the sensitive content.
The answer lies in this GitHub help post https://help.github.com/articles/remove-sensitive-data. However, the post missed a few points which had me chasing myself in circles as kept re-committing my config file from my working folder in a recursive nightmare. Hopefully what I learned and captured here will save you or someone else some valuable time.