For example if someone said unliterate instead of illiterate or discluded instead of excluded? Is there a word for this kind of grammatical error?
Some sources distinguish between "barbarisms" and "solecisms", using the former for errors in morphology and the latter for errors in syntax. So, using a nonstandard prefix would be a "barbarism". Insofar as "barbarism" is especially used for intrusions of one language into another, unliterate is a pretty good example (since Latinate roots usually prefer Latinate prefixes like in-, though there are plenty of exceptions like unstable).
Incidentally, disclude is a complicated example, because it used to be standard, and in fact, it's directly borrowed from a regularly-formed word attested in Classical Latin. The Oxford English Dictionary notes only that the relevant English sense is "Somewhat rare and nonstandard since 18th cent." So, be kind to the barbarists; someday your usages, too, will have passed.