The Oxford Dictionary Online (now Oxford Languages)
says that the phrase could use either wrack or rack. They note that
The relationship between the forms rack and wrack is complicated. The most common noun sense of rack, ‘a framework for holding and storing things’, is always spelled rack, never wrack. In the phrase rack something up the word is also always spelled rack. Figurative senses of the verb, deriving from the type of torture in which someone is stretched on a rack, can, however , be spelled either rack or wrack: thus racked with guilt; or wracked with guilt; rack your brains; or wrack your brains.
However, according to this entry for wrack in EtymOnline, the term should be rack:
The verb meaning "to ruin or wreck" (originally of ships) is recorded from 1560s, from earlier intrans. sense "to be shipwrecked" (late 15c.). Often confused in this sense since 16c. with rack (1) in the verb sense of "to torture on the rack;" to wrack one's brains is thus erroneous.
The PhraseFinder agrees that the phrase is rack your brains, adding:
The rack was a mediaeval torture device. The crude but, one presumes, effective racks often tore the victim's limbs from their bodies. It isn't surprising that 'rack' was adopted as a verb meaning to cause pain and anguish. Shakespeare was one of many authors who used this.
Further, this book on common English errors says it should be rack:
If you are racked with pain or you feel nerve-racked, you are feeling as if you were being stretched on that Medieval instrument of torture, the rack. You rack your brains when you stretch them vigorously to search out the truth like a torturer. “Wrack” has to do with ruinous accidents, so if the stock market is wracked by rumors of imminent recession, it’s wrecked. If things are wrecked, they go to “wrack and ruin.”
The Grammarist agrees it should be rack as well.
That being said, there is some use of wrack your brains (blue line) as shown by this Google NGram. However, rack your brains is correct and more common:
There is some disagreement, but more sources say it should be rack. Thus it seems more likely that the phrase is rack your brains instead of wrack.