How about **granular**?

OD:

> [**granular**](https://en.oxforddictionaries.com/definition/granular):
> having a roughened surface or structure.

Your examples:

> The **granular** cover of the book.
> 
> The **granular** surface of the wall.

Another possibility is **matte**. From [Wikipedia](https://en.wikipedia.org/wiki/Paint_sheen):

> In paint technology, the sheen is the glossiness of a paint finish.
> Glossy and flat (or **matte**) are typical extreme levels of
> glossiness of a finish. Glossy paints are shiny and reflect most light
> in the specular (mirror-like) direction, while on flat paints most of
> the light diffuses in a range of angles. The gloss level of paint can
> also affect its apparent colour. [**emphasis added**]
> 
> Between those extremes, there are a number of intermediate gloss
> levels. Their common names, from the most dull to the most shiny,
> include: matte, eggshell, satin, silk, semi-gloss and high gloss.
> These terms are not standardized, and not all manufacturers use all
> these terms.
> 
> The sheen or gloss level of a paint is principally determined by the
> ratio of resinous, adhesive binder which solidifies after drying, and
> solid, powdery pigment. The more binder the coating contains, the more
> regular reflection will be made from its smooth surface; conversely,
> with less binder, grains of pigment become exposed to the surface,
> scattering the light and providing matte effect.

The main idea here is that **matte** finishes are rougher than glossy finishes, which is why they reflect light non-specularly and feel slightly rougher than glossy finishes.

Your examples:

> The **matte** cover of the book.
> 
> The **matte** surface of the wall.