Here are the two main bits of positive evidence that to is part of the infinitive.
1. It doesn't show up with nouns in the same situations.
- I like to dance.
- I like pizza.
Notice that to is not an inherent part of like, because even in the exact same sense of the word, if it is followed by a noun there is no to.
2. It shows up without another verb.
In phrases without a verb in front of the to, you can see it appearing:
For modals, as others have explained, the modal verbs do something special with the verb, as do the helper verbs "to be" and "to have". But they are a special case. Modal verbs also don't allow a gerund after them in any situation, e.g., you can't say "*I would walking" like you can with a verb that is usually followed by to. But with "like" you can say "I like to dance" and "I like dancing". Only the form of the verb is changing, from the infinitive to the gerund.