(I’m not sure if I’m repeating, agreeing, or disagreeing with what Edwin’s answer says… but I have some different points, too.)
There are various types of elements that can be used to modify nouns: possessives (arguable whether they modify or merely determine), other noun phrases acting as adjuncts, adjectival phrases, and verb phrases, to name a few. Each of these can modify in two ways: attributively and predicatively.
In predicative modification, a finite verb is required in English, and with types that don’t already supply one (i.e., everything except verb phrases), a dummy auxiliary BE is inserted.
In attributive modification, it’s the other way around: a finite verb is not allowed. This isn’t a problem for possessives, noun phrases, and adjectives phrases—you just plop them up before the noun they modify and Bob’s your uncle. With verb phrases, you need to do a bit of moving around, though. You can do that in two ways: by moving the verb phrase into a subordinate clause which follows the noun phrase, or by creating an actual attributive verb phrase which precedes the noun like an adjective or noun adjunct.
In order to satisfy the no-finite-verbs rule when creating this type of phrase, you convert the finite verbal form into a corresponding participle: present participle if the finite verb is active, past (passive) participle if it’s passive. If the verb in question has no complements, the participle on its own modifies the noun phrase and is traditionally often called (and often fairly indistinguishible from) an adjective.
If the verb does have a complement—meaning for transitive verbs essentially an object, for linking verbs essentially a subject complement—the entire verbal phrase is turned into a compound. In a verb phrase, the verb is always the head and the complement is the dependent, and this relationship is maintained when the phrase is turned into a compound: the participle (verb) becomes the head and the complement becomes the dependent.
In your example here, look is a linking verb, and interesting is its (subject) complement. The underlying—if you can call it underlying—predicative construction to “[this is an] interesting-looking book” is “[this] book looks interesting”, where looks is the finite verb and interesting is the subject complement. With a transitive verb, you might turn “the woman runs marathons” into “the marathon-running woman”.
In this scenario, calling the participle an adjective is more problematic. Adjectives generally do not take complements, and on the rare occasion that they do, they cannot be used attributively in this manner with their complement. To use the most well-known example of a ‘transitive adjective’, worth, you cannot transform “the book is worth money” into “*the money-worth book”. You can, however, turn “the book costs money” into “the money-costing book”, even if it’s not exactly pretty or idiomatic. If you want to shoehorn participles into the group called ‘adjectives’, you will at least have to make them a special subgroup of that group, with special properties not shared with other adjectives.
The modifier ‘interesting-looking’ is an endocentric compound made from a verbal phrase (what Germans calls a verbales Rektionskompositum and English sadly doesn’t really have a standard term for), and it possesses the general characteristics of endocentric compounds, one of which is that the dependent modifies the head. So interesting (the complement/dependent) modifies looking (the verb/head). This is another way in which participles differ from true adjectives: adjectives cannot modify true adjectives, but they can modify participles.
(As for ‘looking glass’, that is an altogether different kettle of fish. Looking here is not an adjective, but also not a participle: it’s a verbal noun… or possibly a gerund. I don’t think there’s really any way to tell which. Traditional grammar seems to label -ing forms found as the first member of compounds as gerunds. At any rate, it’s a noun, used as a noun adjunct to modify the head glass in another endocentric compound. A looking glass is not a glass that is looking [at something], but a glass for looking.
The reason that looking on its own is not normally used to modify a noun phrase, though, is because it’s not normally used as an intransitive verb either, except in the imperative. That doesn’t mean it can’t be used this way, though: “the man looks” would be transformable to “the looking man”.)
