I'm surprised no one has mentioned what khakis actually means in some non-rhotic accents. Non-rhotic accents are ones in which the r sound is only pronounced when it's followed by a vowel (e.g. right); the r is silent elsewhere. Khakis is a humorous pronunciation-spelling of car keys in some areas of the USA (such as Boston). The r is silent because it's followed by a consonant, so it's pronounced KAH-KEEZ which could alternatively be spelt khakis.
Here's an illustration from A Historical Phonology of English by Donka Minkova (p274):
I assume it also means car keys in the quote in the OP:
You are not the car you drive
You are not your fucking khakis! [car keys]