Which is more grammatically correct?
London's first political magazine
London's first politics magazine
Technically for either to work, one of the last two words has to be acting as an adjective (modifying the other).
The word political is an adjective (and sometimes an adverb), so no problem with your first sentence.
The second is more problematic. It looks (and sounds) awkward to my ears, as neither word is typically used as an adjective. Whichever one is the noun has to also be modified by "first", so I guess politics is out (although a magazine about the early politics of London back at its founding might be interesting to me...), and thus must be the other adjective. However, I can't think of any other common phrase where it is used that way. Merriam-Webster defines it only as a noun.
So I'd say the first is much better than the second.
They are both grammatical.
It is difficult to explain the differences without the context, but I suppose a politics magazine might be one about politics and a political magazine one that has a particular political axe to grind. Very much as Jim has said in his comment, in fact. However, the question is about grammaticality, and that was the question I answered.
Political magazine: A magazine that is engaged in affecting political processes, such as by encouraging readers to espouse a particular political view, as in "The magazine is political."
Politics magazine: A magazine about politics.
Different meanings, both grammatical.