It appears that there is no specific origin of the phrase as suggested by The Phrase Finder :
'There's a fine line between...' means that the two things mentioned are very close. For example, there's a fine line between genius and madness, or love and hate.
These fine lines may be the origin of the phrase, or at least the source of its popularity.
"Great wits are sure to madness near alli'd
And thin partitions do their bounds divide."
John Dryden 1631-1700 in Alexander's Feast 1.163
Ngram suggests that the phrase became more and more popular from the beginning of the 20th century.
One of my favourite quotes:
“a thin line separates love from hate, success from failure, life from death, a line as difficult to walk as a razor’s edge”. ( W. Somerset Maugham)