Having been a non- native speaker, several times I had been asked to write my name IN VERNACULAR or IN MOTHER TONGUE during my college days. I couldn't tell apart the subtle difference and thought of them to be synonymous those days.
According to the Wikipedia article, VERNACULAR is the language spoken by a group of people inhabiting a particular region/ country which is regarded as of low - status.
Mother tongue is in some countries the language/ dialect of one's ethnic group rather than the first language — the one exposed to, from his birth.
Thus, to reach a conclusion,
The definitions of both terms overlap to a certain degree.
The definition of mother tongue seems little VAGUE.
Which out of the two is MORE REFINED & FORMAL to denote one's first language? I ask this on the grounds:-
The term vernacular has derogatory connotation.
On the contrary, I think that mother tongue is a SUBSET of vernacular and not vice versa. (In most cases, one's mother tongue is the product of their vernacular.)
Also, I'm personally of the opinion that if somebody asks you to use your vernacular, it indicates imperialistic approach of the instructor as witnessed by the excerpts given below. (I'm taken by the retro-noir-word "ANGLO-VERNACULAR").