Using parentheses is generally considered bad form or lazy. Usually you'll find that their contents were unnecessary, or could've been more clearly stated with a rewrite. The best way to write the sentence would be like this.
This car has an optional navigation system.
However, if you are just asking about how to use them, then the first sentence is correct, even though it looks strange.
This car has a(n) (optional) navigation system.
Some stuff on parentheses. http://www.writingsimplified.com/2009/04/how-and-when-to-use-parentheses.html