Yes, it's ungrammatical in formal usage. It means "the ball doesn't lie", but the grammar makes it sound street-tough and world-wise, .
It's an Americanism, possibly AAVE derived, with the word "don't" taking on a generic "do" role, much as "ain't" takes on a generic "is" role in rural speech. Compare
- Ball ain't bouncing.
- Ball don't bounce.
- She ain't seen me.
- She don't see me
- Ain't he there?
- Don't he hang there?
I wouldn't call these ungrammatical, although they are in formal contexts. They are particular to a street or youth idiom. It's a reduction in the number of conjugations required, making the sentence form regular across different pronouns. It's the same form with "he", "you", "I" in the subject role.
I found it in an AAVE feature list, it's 3rd singular don't on the list. But it doesn't feel specifically like AAVE, because it's not complex enough to be a unique AAVE thing, like "That ball done been lying" would be.