Your first two examples are a special use of of that's not readily explained by reference to its other uses. In each of them, the of is optional ("more of a sanity check" = "more a sanity check"; "more of a hack" = "more a hack"), and serves to introduce a singular countable predicate noun that's modified by more. The same happens with much ("it's not much of a problem"), less ("Would you think me less of a man?"), and a few other adverbs of quantity. It doesn't generally happen with non-count nouns, nor with plural nouns; *"they're more of sanity checks than anything" and *"there's not much of reason" are both ungrammatical, or at least, very awkward. Even with singular count nouns, I think this use of of may be specific to certain dialects; I (an American) find it completely normal, but I've heard Britons describe it as a strange Americanism.
Your third example is different; it's one of the ordinary uses of of, where it links together two nouns. The core of the sentence is, "A course would be of utility", which means "A course would be useful."
"It's more like a sanity check than anything" is grammatically correct, but would not have quite the same meaning. If you wanted to rephrase it, I think the safest rephrasing would be, "It's a sanity check more than anything."