While Grammarly bills itself as the "World's Best Grammar Checker," it has limitations. In fact, it must play the law of percentages, and - like many grammar checkers - fails to happily marry semantics and syntax.
What I mean is that "custom" is, in this case, identified as a noun. Grammarly doesn't like to see "custom" plus another noun, despite the fact that many dictionaries do in fact list "custom" as an adjective, with common examples being "custom suit" or "custom cabinets." Try those expressions (undeniably common ones) in Grammarly and see what happens. They will be flagged as noun problems.
Grammarly wants the adjective "customized." However, with fabricated objects we typically use "custom" rather than "customized," just as in your first example. However, some people may find "custom solution" a bit awkward, because with solutions and services and non-fabricated items it may be more common to use "customized."
Note that Grammarly doesn't recognize all noun + noun combinations as incorrect. For example, type "I have a grammar problem" into Grammarly and it will pass.
p.s. what the heck word form is "Grammarly" anyways?