I usually start with what I'm focusing on as the "main" flow of the song. If I'm doing a song that the drums are my main focus, thats where I start. If I'm doing a song that the piano or melody is my main focus, thats where I start.
I think that most producers start with the drums because the drums usually set the tempo of the song and make it easier to build on AND the drums is one of the few instruments that USUALLY play from the begining to the end without stopping.
I don't think that there is any TRUE format for producing a song (i.e. you have to start with drums, then you have to do your melody, yahda, yahda...) I think its all up to the individual.
--dac