Good question. Actually, in certain contexts you can say things like that...
"Hey! You wanna come bungee-jumping this weekend?"
"No way! It's far too dangerous!"
"I'm much too careful to trust my life to a bit of elastic cord!"
"Wimp! You're so careful you wouldn't cross the road without the lollipop lady to hold your hand!"
Grammatically there's nothing wrong there. It's just that with the particular activity being careful and infinitive verbs, we're more used to constructions like...
1: I'm always careful to avoid judging a book by its cover.
2: The burglar was careful to avoid leaving any fingerprints.
3: The burglar was too careful to leave any fingerprints.
Notice how in #3, leave any fingerprints appears syntactically identical to OP's go across the road. But actually, there's a big difference. We can recast OP's example into
You can't be too careful [when] going across the road.
(The more careful you are, the better. No matter how careful you are, you can't be too careful)
But with my example #3, we simply can't introduce the gerund like that...
3a: *The burglar couldn't be too careful leaving any fingerprints.
...where I don't think any amount of "contrived context" would make #3a acceptable to native speakers. The difference is down to semantics...
to go/going across the road - is normally an activity that can be done with more or less carefulness.
to leave/leaving fingerprints - is a "careless" activity (doing it at all indicates lack of carefulness).
I'll leave it to wiser heads than mine to explain exactly why a "contrived context" would be (just) possible in #3a, if I didn't have the word any (but becomes totally impossible when it is present).
TL;DR: OP happens to have chosen a specific activity (being careful [when] crossing the road) that conflicts with the normal context where we talk about being too careful (to take risks).