The correct construction is to be left free, meaning that the subject is free of outside interference.
Her boss had a piercing headache, so the employee was left free to solve the computer error herself.
Left in this construction is the past participle of leave, with the meaning of allow, permit, or, interestingly enough, let. Unlike the many other common constructions involving leave, this one is unique - most other constructions imply that the subject is alone whereas with this construction that isn't necessarily the case.
Your other alternative, to be let free is grammatical, but unusual to my ears. It either needs an auxiliary verb and object (to let something be/go free) or a different primary verb (to be set free) to sound more accurate. Contrary to the usage with left discussed above, these mean that the subject was released from some bondage or constraints, and not that they were free of outside interference.
After its non-stop barking, the dog was finally set free.
After ten years in prison, the wardens let me go free.
A brief assessment of the sentences you provided:
Leave me free to do whatever I want. Ungrammatical (wrong formation of imperative of passive construction)
I was left free to do whatever I wanted. Correct (implies lack of interference)
Let me free to do whatever I want. Ungrammatical (needs be)
I was let free to do whatever I wanted. Okay but weird (implies release from constraint)