It's important here to distinguish between the preterite form of a verb, and the past tense meaning that is normally associated with the preterite form. The modal preterite use of some English verbs indicates modality, as the OP points out (not past time). For example, in
I wish you went to the doctor more often than you do.
If you went to the doctor tomorrow, I'd drive you.
...use of the preterite form of the verb (went) does not indicate past time.
To answer your question (why you can't say "If I were you, I went to the doctor"), English does not grammaticalize a generic concept of modality (nor does it have a robust grammatical category "subjunctive" as French does), but instead has several common constructions that can be used to convey different types of modality. The modal preterite construction cannot be used in the same situations as a modal auxiliary can. The modal preterite expresses modal remoteness, generally in one of the two types of constructions:
- Conditional constructions. (e.g., If she injured herself, I'd rush to help.)
- Expressing a counterfactual situation, but only in subordinate clauses which complement specific verbs that have an element of uncertainty in their meaning (e.g., wish, think).
Auxiliary verbs can be used to convey a larger range of modal meanings (with different auxiliaries having different modal meanings associated), some of which include remoteness. In a sentence like "I would go to the doctor (If I were you)," would expresses two types of modality: propensity (lexically) and remoteness (preterite form).