In short, both are correct. Though in the purest sense, it could be that neither are.
Most dictionaries seem to contain only the classical definition ("a disturbance or fuss") rather than the modern one that most people use today ("something to be completed"). I was unable to find any "task" definition other than in Wiktionary and The Free Dictionary; at least online, it's missing from the New Oxford American Dictionary, Collins English Dictionary, Chambers 21st Century Dictionary, Merriam-Webster, Cambridge Online Dictionary, and Longman Dictionary of Contemporary English. Can anyone confirm this?
According to the Online Etymology Dictionary, To-do came into use in the 1570's, meaning "proper or necessary to be done". A newer meaning emerged in the 1820's, being "a disturbance or fuss". I saw someone quote a 1920's figure for a resurgence of the "task" meaning, but according to the N-Grams it looks like that didn't happen until at least the 80's.
The few dictionaries that recognize the "task" meaning generally consider both to-do and todo as proper. Though as others have noted here, to-do is vastly preferred by formal publications; only in the programming world and some informal contexts does todo seem to be more common.