I would like to explain his whole quote.
First, the sentence
The programmer, like the poet, works only slightly removed from pure thought-stuff.
By this, the writer means, that similar to a poet, the job of a programmer is Almost based on pure imagination. The work of a Programmer is not Totally dominated/based on pure imagination but is Almost based on pure Imagination/thought-stuff (that is, it is slightly removed from it). He is not talking about the difference in a poet's work to that of a programmers, but rather about the similarities in the sense that how imaginative they are (while coming just short of purely imaginative/thought-stuff).
His next sentences beautifully follows from his previous assertions.
He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures.
The programmer "builds his castles in the air, from air, creating by exertion of the imagination"
I have always felt the same about programming (being a programmer myself). Programming is vastly different from other forms of Engineering (I consider programming to be an Engineering vocation). Unlike Civil, Electrical or Mechanical Engineering where the creator (Engineer) is severely limited by the physical, material and other 'practical' constraints to truly create whatever they have the ability to imagine (think warp-drives!), the programmers not so much.
Programmers are largely unshackled by the limitations of the physical world. They have both the ability and the means to conjure up extremely complex, grand and creative structures from nothing but figment of imagination as if they are "building a castle in the air, from air, created by exertion of imagination". And as I mentioned like some of the other branches of Engineering "few media of creation are so flexible, so easy to polish and rework." And if some of those media does permit such freedom of imagination in their implementations (painting/poetry), they hardly ever impact the physical world in the same ways like programming do.