From a post on gamedev.net:
there was a time when "down to the metal" meant basically writing code
for the raw hardware (with no OS or device drivers involved), and
often with stuff written in a fair bit of assembler as well.
"Metal" refers to the physical hardware, trying to cut through the driver or operating system levels, i.e. low level CPU, graphics programming, real-time embedded systems. Some call themselves "down-to-the-metal C++ programmers".
It might be related to the expression: "pedal to the metal", where (gas) pedal is fully depressed to the car floor, providing as much speed as possible, which is one of the purposes of "to the metal" programming. Additional shades are given in Close To The Metal, referring to code limited to some hardware or security issues..