In computer image processing, it's probably a bad idea to use mat or matte to refer to a border, because of likely confusion with another use of matte. For example, I was unaware of this sense of mat (OED: "an ornamental (often gold) border of paper, metal, etc., surrounding a framed picture. Later also: a border or mount of white or tinted cardboard").
The other sense of matte comes from film-making and means a cut-out that hides parts of an image. In the days of physical film, a matte would prevent parts of the emulsion from being exposed, allowing other images to be exposed on those parts later.
(The OED notes both spellings mat and matte, but you can see by comparing Google Books searches for "a mat shot" and "a matte shot" that the latter is the overwhelmingly more popular spelling.)