Which of the following is considered correct? ". . . where I was at age five." or ". . . where I was at age 5."
This would be a matter of style. Different style guides and audiences expect different formats. For example, some style guides specify that numbers under 100 be actually spelled out ("ninety-nine"). Other style guides do not require this. I would make sure you know who your audience is and what they expect to see.