The subject, "it", seems ambiguous.
Interpretive possibilities follow, and include spoilers
1) "It's all in your head" - The act of joining Slytherin would make Harry Potter great (it), and is already set up in his head.
The idea that Harry should have joined Slytherin is referenced from the beginning, from his ability to speak that snake-tongue thing to... well, read / watch the last book / film. He's a perfect fit for Slytherin.
2) "It's all in your head" - The possibility to be great (it), if joining Slytherin, exists within Harry's head.
Most of Slytherin aren't nice; that's why they're associated with snakes. By making this suggestion, the hat suggests that Harry possesses the power needed to harness the Dark Side of the Force, so to speak. The Hat makes no comment in this part of the diagnosis about Harry's ability to be great elsewhere. Again, Harry's dark side is referenced a lot.
3) "It's all in your head" - Another house might make Harry great, but he would have to change, whereas the forecast for greatness with Slytherin (it) is already in his head, without the need for help.
Most of the people on Voldemort's side rely on their own power and instinct, rarely trusting anyone else. In contrast, Harry relies a lot on his friends. The hat asserts that he would be successful in and of himself with Slytherin, without making the same guarantee for other houses. Harry's choice, to rely at least in part on others, shows how socially-minded he is (ergo, not evil).
4) "It's all in your head" - Slytherin delivers some core elements that are needed to fulfil Harry's greatness (it).
Without people from Slytherin, Harry was always doomed to failure. Those elements were eventually provided by others.
I think that the use of the phrase was deliberately or instinctively ambiguous, and that Rowling was a better author than I thought she was before I looked properly.