*Not bad an attempt is in fact not a grammatical noun phrase.
English noun phrases can have both articles and adjectives, but the articles have to come before the adjectives. An is an article and bad is an adjective. So the order has to be changed.
- Not a bad attempt is a grammatical noun phrase.
Not bad is grammatical by itself, but only when it's a predicate adjective that follows an auxiliary verb, like
The not is part of the verb phrase is not bad here.
But in the ungrammatical *not bad an attempt, not bad doesn't form a constituent because it's not part of a verb phrase, and so there's no reason to move the adjective bad in front of the article an.