How about granular?
OD:
granular:
having a roughened surface or structure.
Your examples:
The granular cover of the book.
The granular surface of the wall.
Another possibility is matte. From Wikipedia:
In paint technology, the sheen is the glossiness of a paint finish.
Glossy and flat (or matte) are typical extreme levels of
glossiness of a finish. Glossy paints are shiny and reflect most light
in the specular (mirror-like) direction, while on flat paints most of
the light diffuses in a range of angles. The gloss level of paint can
also affect its apparent colour. [emphasis added]
Between those extremes, there are a number of intermediate gloss
levels. Their common names, from the most dull to the most shiny,
include: matte, eggshell, satin, silk, semi-gloss and high gloss.
These terms are not standardized, and not all manufacturers use all
these terms.
The sheen or gloss level of a paint is principally determined by the
ratio of resinous, adhesive binder which solidifies after drying, and
solid, powdery pigment. The more binder the coating contains, the more
regular reflection will be made from its smooth surface; conversely,
with less binder, grains of pigment become exposed to the surface,
scattering the light and providing matte effect.
The main idea here is that matte finishes are rougher than glossy finishes, which is why they reflect light non-specularly and feel slightly rougher than glossy finishes.
Your examples:
The matte cover of the book.
The matte surface of the wall.