The first form "If you were to go home, you would feel better." should be grammatically correct, but it sounds rather strange to me.
The second form "If you went home, you would feel better." is grammatically absolutely correct and also expresses the right thing. It is a so-called Conditional Clause of Type II which means that the event in question (i.e. you go home) is improbable but still possible. In general such a clause is constructed according to the pattern: If + simple past, would/could/might + infinitive.
The third form "If you will go home you will feel better" is incorrect. If you slightly adjust it to "If you go home, you will feel better." you get a so-called Conditional Clause of Type I which expresses that the event in question is likely to happen. In general a Type I If-clause follows the pattern: If + simple present, will-future or can/must/might+infinitive or imperative.
There is also a Type III, which, in your case, would be "If you had gone home, you would have felt better." It implies that the event in question is impossible, because you are talking about the past. In general, Type III follows the pattern: If + past perfect, would/could/might + have + past participle.
Other conditional if-clauses that do not fall into one of the above categories are usually grammatically incorrect. As always, there might be some exceptions and special cases, but the above is definitely a good guideline.
EDIT: People also sometimes speak of a Type 0 if-clause which addresses something that is generally true, for example: If it rains, I take out my umbrella. The construction is fairly simple, as you see.