I'm surprised no one has mentioned what khakis actually means in some non-rhotic accents. Non-rhotic accents are ones in which the r sound is only pronounced when it's followed by a vowel (e.g. right); the r is silent elsewhere. Khakis is a humorous pronunciation-spelling of car keys in some areas of the USA (such as Boston). The r is silent because it's followed by a consonant, so it's pronounced KAH-KEEZ which could alternatively be spelt khakis.
Here's an illustration from A Historical Phonology of English by Donka Minkova (p274):
I assume it also means car keys in the quote in the OP:
You are not the car you drive
You are not your fucking khakis! [car keys]
Maybe I have a clue. Sir Terence Conran, a founder of Habitat, an advocate of design and minimalism, supposedly said, "I am not my khakis"
The stuff you own, wear, etc. does not define you. This is not about a higher status symbol. This is just about things / stuff in general. I suppose the minimalistic rhetoric of Fight Club refers to this sentense, because it is a symbol of not beeing addicted to material posessions.
The character Tyler Durden is referring to a popular commercial (late 1990's) from The Gap clothing company. It showed people dancing to swing music (a retro trend at the time) to promote their khakis pants. His line is mocking the vapid materialistic commercialism of the era. Which is sadly tame compared to now in 2021.