It's a way to emphasize that he hates to leave it. It suggests that that may be good reason reason for him to be elsewhere, but he just doesn't want to leave where he is.
Many times people will focus on one aspect of a situation and make that the basis of a decision, and say something like, "is all" or "That's all I'm saying." For example,
A: I think you should go outside. You need a break.
B: Yes, but I really have to finish this.
A: I'm not saying you shouldn't finish, but I think you'll do a better
job of it if you take a break, is all. [or: "That's all I'm saying."]
There are several versions of this. Sometimes people will say:
You could also use "The thing is" or "it's just that":
I know I'm late, but the thing is, I just hate to leave it.
I know I'm late, it's just that I hate to leave it.
which means "I know I must go, but I don't want to--not because I don't want to be there, but because I don't want to leave here." This is the same as:
I just hate to leave here, is all.