"I have nearly finished..." is in the present perfect tense, which is used to talk about actions that began in the past and ended in the past or the present but still have a bearing on the present.
"I have nearly been finishing..." is in the present perfect progressive tense, which is used to talk about continuous actions that began in the past and continue to the present.
For example, "I have read three poems" means that I read three poems in the past. "I have been reading three poems" means that I started reading three poems in the past, and I am still reading them in the present.
Now to your example: it seems unorthodox to put "finish" in the present perfect progressive tense because to finish something is a finite action, not a continuous one. For example, you may work on something for a period of time (a continuous action) but you finish it at one point in time (not a continuous action). The only reason to put a verb like "finish" in the present perfect progressive tense is if you are deliberately stressing that someone took a period of time to do something that should have been done at one point in time. For example, "John constantly procrastinates; he has been finishing his essay for three hours" means that John should have finished his essay at a single point in time, but due to his procrastination he has been dragging the process out over a period of time.