I suspect that you're referring to what originated as a form of abbreviated common messages in radio communications. Starting in the early 1900s, "Q" codes were developed, a collection of standard questions routinely used in maritime and aeronautical situations. They were given three letter codes, all beginning with the letter Q. You can find information on Q codes in this Wikipedia article.
Two of the Q codes deal with the quality of the communications link:
QRK: What is the intelligibility of the signal?
QSA: What is the strength of the signal?
Each has five possible responses:
The response to the QRK/QSA questions would be two numbers, the first being the intelligibility or readability score and the second being the signal strength score. So "five by five" would mean the best possible signal. More background on this can be found in this Wikipedia article. If you found the reference in an air force pilot training manual, that would make sense.
The expression has worked its way into more popular use. "Five by five" (occasionally written as variations such as "5 × 5", "five by", or even just "Fives"), has come to mean "I understand you perfectly" in situations other than radio communication. Post-World War II, the phrase "loud and clear" entered common usage with a similar meaning.
For situations that only concern intelligibility and don't involve radio signal strength, the second number is irrelevant, anyway, so "five by" replicates the style and is the equivalent of what in the current vernacular would be "five by whatever".
In the TV series "Buffy the Vampire Slayer", one character used "five by five" as a sarcastic catchphrase.
From the question, it would appear that other variants, like "three by" may also have found their way into general usage or slang. I couldn't find any references to "three by", but the Q code responses would suggest that it may mean something like "I'm having trouble hearing you", which would be an apt description of some of my cell phone reception.