One could grow weary of pointing out that English doesn't actually have many "rules", and that it would be better in most cases to call potential candidates for that term "strong tendencies", since they're often not universally applicable.
OP himself may be exceptionally consistent in using rather rather than quite large, but that's really just his preference. Though if this NGram means anything, it suggests average usage is tipping the other way. He's more in the mainstream with quite amazing, but this one backs up my own gut feeling that pretty soon pretty amazing will rule that particular roost.
In certain contexts, these kind of qualifiers can all have their own special nuances, but it would be a mistake to infer any consistent rule regarding which to use when. Nor is there any consistent hierarchy of "intensity" for rather, quite, pretty, very, etc. Many Brits, for example, would say that in some contexts, somewhat is far more intense than any of those. Any many (younger?) people use totally in ways that imply it's far less intense.