This is a difficult one, even for a native speaker - this would usually be done unconsciously, but here is how I would use each preposition:
"For me ..." suggests a consumable, like coffee, or a favourite dish, and describing the taste, or lack of taste. If someone were to ask: "Which lettuce do you prefer in your salad, John?" The reply might begin: "For me, the crispiness of Romaine gives it a better taste..."
On the other hand, "To me ..." suggests possessions, or things in general, or something abstract, or even if no opinion is expressed. Very often, you'll see this at the end of a sentence:
"Which airline company do you prefer, Sally?" The reply might go like this:
"You know what, none of the airlines is really interested in the customer, they're all the same to me,"