The English language has been primarily spoken either by Christian people or by people who are familiar with Christian concepts since part-way through the Old English period (after the Angle-Saxon people were converted). For that reason, Christian concepts have had a strong influence upon the language in a variety of ways both subtle and blatant, including ironically in most of its stronger terms (to use the concept of damnation in such a manner is arguably blasphemy, and hence a sin).
Mainstream Christian views of the afterlife offer the alternatives of Salvation in which one ends up in heaven, or Damnation in which one ends up in hell, with both being for all eternity (purgatory and limbo complicate the matter somewhat but aren't relevant). There are other Christian views, but most of them are relatively recent.
Hence, go to hell, damn you, damned, darn (a variant of damn considered less forceful), to hell with you/it/them/him/etc., devil take him (dialect de'il take him relatively common at one point) and other expressions conveying that you hope for, or expect, the damnation of the subject are quite common.
In usage they are considered mild expletives - not very polite, but less offensive than the sex-based expletives.