In the sentence:
How deep or deeply should I study something?
Which of the two is more appropriate?
In that context, it's deeply.
Deep is an adjective, so can only be used to describe the quality of a noun:
How deep is the water?
Deeply is an adverb, so can only be used to modify a verb:
Were you thinking deeply?
Deep is often used instead of deeply, to the extent that it is largely accepted without comment. However, I've never heard it the other way around. "What a deeply question" is just plain wrong.
Deeply is probably better in this context, because it's an an adverb of manner or degree, and is commonly used as an intensifier. In this case it can indicate the degree or intensity of the studying.
Deep, when used as an adverb, is usually used as an adverb of place.
While deep can also be an adjective, deeply, of course, is not used as an adjective.
This is not necessarily an exact science, and it's certainly not a question of formality or informality. In fact, there's an argument for "How deep should I study something" if deep is thought of as resultative – i.e. as a result of studying something, I will be deep inside of it, which is an expression of metaphorical location. However, a similar concept can be expressed with "deeply immersed in it", which may seem at first blush like location but is probably better characterized as an expression of the intensity or degree of the immersion.
In the end it may be a question of whether it's the intensity or degree of the verb process or the locative result of the verb process that's being emphasized.