This website says
It's the result of the same process (that is, erroneous pronunciation) whereby "learn" becomes "larn" in some (very) nonstandard American dialects. One feature of uneducated speech in England around the 1800's was a tendency to pronounce the "er" sound of words like "clerk" as the "ar" sound of "clark". The phenomenon was sufficiently widespread that the English novelist Henry Fielding used pronunciations like "sarvis" for "service", "sartain" for "certain", and "parson" for "person" in the speech of characters meant to seem vulgar or unintelligent. Due to the overwhelming influence of such people in England (that is, the uneducated), these previously unacceptable pronunciations eventually became standard for some words, like Derby, Berkeley, and clerk...
J.C. Wells, Accents of English
So a more general sound change that took "er" to "ar" was present in some lower-class British accents when Henry Fielding wrote Tom Jones (ca. 1750). This 1849 dictionary found in Google books indicates that this pronunciation had for the most part vanished a hundred years later1. The pronunciation now persists in only a handful of words:
Berkeley, clerk, derby, Hertfordshire, sergeant, varsity.
I take the liberty of including varsity here because it was originally short for university. However, university has reverted to the standard pronunciation, while varsity has switched to the phonetic spelling.
1 Interestingly, the dictionary also says that many Americans pronounced sergeant with "er".