There is no difference in meaning between the first two sentences:
- If you saw a lion in a thick forest, what would you do?
- If you were to see a lion in a thick forest, what would you do?
Your third sentence is also grammatical, but means something slightly different:
- If you had seen a lion in a thick forest, what would have you done?
The difference is that the first pair are talking about a hypothetical situation in the future, so something that has not yet occurred, while the third is talking about a hypothetical situation in the past, that is, one construed to have already occurred.
Your fourth sentence is not grammatical in English:
- If you *would have seen a lion in a thick forest, what would have you done?
It's not grammatical because in English the protasis (the first part) of a conditional must not itself be in the conditional; that mood-like sense is reserved for the apodosis, the second part, and only when the protasis is in the past. There are many other conditional constructs in English beyond this one, but "If X would ..., then Y would ..." is not one of them. Native speakers of German sometimes try to use this sequence when first learning English, but it is not valid in English.
In contrast to JSBձոգչ's answer,
I don't myself find the were to version as more formal, nor period-limited. Instead, I just see it as wordier than need be. Please note that I am not saying he's wrong, just that I do not share his view. Native speakers often differ on things like this, depending on their own experiences.
One arguable advantage with the were to construct is it does allow for expansion to make things come across as more tentative if desired. For example, If you were to happen to see ... The construction I see as slightly antiquated is If you should see, or its extension If you should happen to see...
A second arguable advantage with the were to construct might be that it somewhat emphasizes the subjunctive / hypothetical aspect of the protasis. That's because were is the last morphological vestige of the Old English past subjunctive that remains in Modern English, and even here is distinguished from the past indicative in only the first and third persons singular.