English is an interesting language in that we have one noun case (I'd call it the Dative, on the basis of Latin cases) where we need to unless we invert word order.
This Dative case is when something is to/for someone (when a noun is an indirect object). He spoke to me; He waited for me. This case always uses a preposition with intransitive verbs (verbs that don't admit a direct object), like the ones above. However, with verbs like 'wrote', you can have a direct object like 'it' or 'that', or noun like 'a book'/'a letter'. When we have a Dative with a transitive, you can say either 'He wrote a letter to me' or 'He wrote me a letter' (inverting the word order to get rid of the preposition).
These phrases are perfectly understandable because 'me' doesn't make sense as the object of write. Grammatically, 'me' on its own is the right case, but semantically, it doesn't make sense - how could someone write you?Because of this, the meaning is perfectly understandable.
When you have a phrase like he gave her it, because there is no preposition 'to', a reader assumes the inverted word order and therefore takes it as the thing being given and her as the person receiving it.
Because 'gave' is transitive, in a sentence like 'he gave her it', inversion makes perfect sense. However, if you don't have an object (it), inversion doesn't work - he gave to her cannot become he gave her as a standalone sentence. However, in a compound sentence of multiple clauses, your object might be stated earlier, which allows for omission: "the advice he gave me was good" makes sense because 'the advice' is introduced as the subject of the sentence, and 'he gave me' modifies it as if it were the direct object of 'gave'. Thus the word order is inverted, 'to' is omitted, and finally the direct object 'the advice' is omitted because it is already implied.
To answer your question, "I remember the advice he gave to me" is correct, and "I remember the advice he gave me" is also correct.