You're referencing an anapodoton. Specifically, "The bigger they are, the harder they fall."
As an intentional rhetorical device, it is generally used for set phrases, where the full form is understood, and would thus be tedious to spell out, as in "When in Rome [do as the Romans]." or "If the mountain won't come to Muhammad [Muhammed will go to the mountain]."
First, the syntax needs to isolate the shortened part so that it is clear that the full quote is implied, or that there's a blank that the listener should fill in. On its own, your original quote doesn't make sense. Here's a possible rewrite using an ellipsis:
- I don't like to be famous. The bigger you are ....
You ask about replacing "they" from the original "the bigger they are ..." with "you" in your quote. However, the syntax suggests that the (second person) "you" should really refer to the first person. If we accept the tuism(?) - and it does sound better in the second person than in the first person - then the switch from "they" to "you" is acceptable in the rewritten form.