It's a jocular archaism. I wouldn't like to say it was ever really spoken in earnest, by a speaker who genuinely feared he was entering land so little known it might contain dragons.
It may have got a bit of extra currency from the tendency for old maps to be "embellished" with drawings. Unexplored areas (land, sea, or unknown) would often be decorated with serpents, and perhaps the legend Here be dragons.
Today it's just a jokey way of saying "There are/may be some really scary things here!". In OP's context, really difficult technical problems.
LATER: Having consulted Wikipedia on this one, I should say the only relevant ancient usage known is the Hunt-Lenox Globe (early 1500s), with the Latin HC SVNT DRACONES (hic sunt dracones, here are dragons) on the eastern coast of Asia.
The earliest reference I can find is this from 1892 claiming cartographers from the 1300s used the phrase, but this NGram strongly suggests the myth didn't become widespread until quite recently.
But it's a nice legend, so I'll keep my "perhaps" at the end of the second paragraph.