"riding all kinds of girls" means, to almost everybody, "having sex with all kinds of girls".
To say that you're "driving all kinds of girls" technically doesn't make sense, either, because the girls are not driveable - they're not cars.
However, "drive" has crossed over into a different, more colloquial meaning, which means "to drive someone else to their destination". The word "ride" has not undergone this transition.
Consider "Can you drive my mum to the airport please?" to "Can you ride my mum to the airport please?". Here, the connotation (in the latter case) is not sexual, but it conjures up an image of someone putting a saddle on the speaker's mum, jumping on, and saying "Giddyup, Barbara!".
In a seperate transition, the word "ride" has also come to mean "have sex with".
So, in the example of "I'll be riding all kinds of girls on my horse", ride (unlike 'drive') has not crossed over to mean "give a ride to", and so we can't take that meaning. Instead, we fall back onto a different alternative meaning, of "have sex with".
Regarding strollers, I think you could say "I'll be giving rides to all kinds of babies in my stroller", although obviously you should always gain permission from their parents.
EDIT as @Kris pointed out in the comments, I didn't answer one of the (several) questions in the Question.
In answer to
"So can you only say:"I'll be giving rides to all kinds of girls on my horse", or is there another way of saying that?"
: it's a weird thing to say, so it's a bit tricky to decide what the best way to rephrase it is. I would probably say "I'll be giving rides to all sorts of girls on my horse.". I think that's the most idiomatic way to say it.