Wikipedia (https://en.wikipedia.org/wiki/Impossible_object) calls them Impossible Objects (which they obviously are) but I have seen another term, possibly from mathematics or psychology, to describe such figures.
Escheresque, especially for art that is inspired by the works of M.C. Escher. -- Wiktionary
Escher + -esque
adj. 1. Resembling the works of M C Escher (1898-1972), Dutch artist, characterised by explorations of infinity, architecture, and tessellation.
Optical illusion is a more general term for all such artwork. -- Wikipedia
An optical illusion (also called a visual illusion) is an illusion caused by the visual system and characterized by visually perceived images that differ from objective reality. The information gathered by the eye is processed in the brain to give a percept that does not tally with a physical measurement of the stimulus source.
Figure-Ground illusions. It's taught in Sensation and Perception courses as such, and often Intro to Psych as well. The word illusion may be dropped from the term.