A reamer is a device or object that enlarges a hole, as this definition from Merriam-Webster's Eleventh Collegiate Dictionary (2013) indicates:
reamer n (1825) : one that reams, as a : a rotating finishing tool with cutting edges used to enlarge or shape a hole b : a fruit juice extractor with a ridged and pointed center rising from a shallow dish.
For its part the verb ream has these meanings (according to the Eleventh Collegiate):
ream vt (1815) 1 a : to widen the opening (of a hole) : COUNTERSINK b (1) : to enlarge or smooth out (a hole) with a reamer (2) : to enlarge the bore of (as a gun) in this way c : to remove by reaming 2 a : to press out with a reamer b : to press out the juice of (as an orange) with a reamer 3 : CHEAT, VICTIMIZE 4 : REPRIMAND — often used with out
What's missing from this description is how reaming in the sense of enlarging a hole applies to human beings. In The Clouds, Aristophanes indicates that in the Age of Pericles, adultery could be punished by subjecting the (male) adulterer to being "reamed with a radish." The technical term for this punishment, apparently, is rhaphanidosis.
Here is a translation (from 2000) of the relevant argument between Superior Argument and Inferior Argument in The Clouds:
INFERIOR. Now, let us move on and discuss the needs of human nature. Suppose that you've been indulging in an illicit love affair. You are discovered! A scandal! What will you do? You are finished because you don't have the means to argue your way out of trouble. But if you choose to make my acquaintance, your nature can run free, with a spring in your step and a smile on your face, and shameful thoughts will never cross your mind. If the husband accuses you of adultery, plead innocence and blame Zeus. Say that clearly he can't resist his lust for women, so how can you, a mere mortal, be expected to have more strength than a god?
SUPERIOR. Yes, but what if he takes your advice and gets punished by pubic plucking, scrotal singeing, and a jolly good rectal radish ramming. No argument of yours is going to help him after that.
The rather vague term "getting reamed" may have its roots (so to speak) in an allusion to this form of punishment, or to something like it.