This sentence
He considered himself unfit for the job, but even more for that task.
does not sound right because unfit has a negative meaning. I tried the other way:
He considered himself unfit for the job, but even less for that task.
but this looks like a contradictory sentence.
Is there any way to put this right, without repeating "unfit"?