"key of" is common when the key is a property of something rather than the key for a lock (whether the lock is literal or metaphorical). So you write a piece of music in the key of C. A really mushy romantic might say something like:
She read the poem in the key of my soul.
In software development we often talk about key/value pairs, which are used in associative arrays and similar data structures, and "key of" would be more appropriate in that context. We also speak about the "keys of" a keyboard.
Speculating a bit, I'll point out that "of" connotes possession, so "key of" makes sense when the key in question is clearly possessed by, say, a piano or an aria. The relationship indicated by "to" is correspondence:
He studied the blueprints to the building.
A classic martini is five parts gin to one part vermouth.
It's easy to see that there's always a correspondence between keys and locks, but we don't think of the key or the lock possessing the other. That's why we talk about the "key to" a particular lock and not the "key of" that lock.