Nigh (archaic) means ‘near’.
As in ‘when winter is nigh’, nearly winter. Or ‘Evening is nigh’ - almost evening.
It still shows up in idioms like ‘nigh on impossible’ as you mention.
But that’s not what I dislike about it, for an academic report or serious piece of writing.
What I dislike about it, is that it’s rather woolly.
‘Almost impossible’
What’s that? Surely if something is ‘impossible’ it means it can’t be done. Adding ‘nigh on’ or ‘almost’ to ‘impossible’ - takes something immutable that definitely can’t be done - an imposing mountain that can never be climbed and adds a woolly little cloud to it that says ‘oooh - maybe!’
Although it is, really, correct in English it just sounds a bit - unscientific.
Instead, how about
- Extraordinarily difficult
- Highly unlikely to succeed
- Phenomenally treacherous
- Outside of predictions
- A fools errand
- A Herculean task
- Monstrously complicated
- Like an ant trying to scale Everest - highly unlikely
In other words, I would explore other more creative ways of painting a picture of the difficulty of the task without resorting to this idiom.