A Collection of College Words and Customs (1851) by Benjamin Homer Hall defines to take the rag off as "to excel; to compose much better than one's classmates." I understand the phrase is quite old; however, I'm curious to see if anyone is able to explain why "to take the rag off" would have meant to excel and to compose much better than one's classmates.
It was common in the 18th and 19th centuries to hold impromptu shooting matches where the target was simply a rag hung on a bush in the distance. A good shot would hit the rag, making it visibly jump. A great shot would literally “take the rag off the bush,” putting an end to at least that round of the contest with an overwhelming success.
Making this sort of shooting match the likely source of “take the rag off the bush” is the fact that it fits perfectly with “triumphant success” sense of the earliest examples we have of the phrase in print. One of these examples, from 1843, specifically refers to a shooting match, and none of them mention religious shrines. There is, on the other hand, no scenario I can imagine involving “rag bushes” that would produce the “stunning triumph” or “take the cake” meanings of “take the rag off the bush.” Finally, although the phrase has been widely used in the US for at least two centuries, it is virtually unknown outside the US.
You can read an interesting debate about its origin here: