'Sit' can have different meanings, one of them being 'to hold a position', which can be an appointed one.
Lexico gives the (weirdly specific) following as one of the definitions of the verb 'sit':
"Live in someone's house while they are away and look after their pet or pets."
Dog-sit is a compound based on babysit, substituting 'baby' for 'dog', while the combining form 'sit' remains (the hyphen has probably been introduced for clarity, either to distinguish 'dog' from 'dogs', or noun from verb).
A similar case is 'dognapping', as compared to 'kidnapping'.
Many similar words are (and can be) formed that way; e.g. 'cat-sit', 'house-sit', &c.
According to Ngram, 'babysit' appeared in (written) American English in the early 1900's, while 'dog-sit' appeared in the late 1960's.