Taking your questions out of order:
What part of speech are these -ing forms.
They are examples of the gerund -- a verb form that functions as a noun. Note well the distinction between "functions as a noun" and "is a noun," as this bears directly on your question. Gerunds are verb forms, but there are also "verbal nouns", which often have the same lexicographic form as gerunds, but are bona fide nouns.
What is the syntactic function of the noun phrases after the -ing forms. (direct object /complement?)
Gerunds can be combined with objects or adverbials, whereas verbal nouns cannot. The noun phrases in question are objects of the gerunds.
The reason that your example "The reading books is prohibited" is ungrammatical is that gerunds do not take determiners, so one of the following must hold:
- "reading" is interpreted as a gerund, and the presence of "the" is erroneous, or
- "reading" is interpreted as a verbal noun, in which case it cannot have an object and the presence of two consecutive nouns is erroneous, or
- "reading" is interpreted as a participle modifying books, and plural "books" does not agree in number with "is".
Whichever you choose, the sentence is erroneous.
Now consider these alternatives:
- "Reading books is prohibited." This sentence can be understood only by interpreting "reading" as a gerund, with "books" as its object, because you cannot have two consecutive, independent nouns there, and, as already observed, "books" does not agree in number with "is". It is thus reading that is prohibited, at least when it is books that are read.
- "The reading of books is prohibited." This sentence can be understood only by interpreting "reading" as a verbal noun. As a verbal form, gerunds do not take prepositional phrases or determiners, but like any other noun, verbal nouns can accept both. This has the same meaning as the previous sentence, but different grammatical form.
- "Reading books are prohibited." This sentence can be understood only by interpreting "reading" as a participle modifying "books", because as a gerund or a verbal noun, singular "reading" does not agree in number with "are". It is thus certain books that are prohibited -- reading books, but not necessarily coloring books.
what part of speech is the word no in examples 1-4?
"No" can be used as a determiner, an adverb, or a noun (at least; Cambridge). These particular constructions are rather idiomatic, which may be why they are troubling you, but I classify the "no" in each of them as an adverb modifying the gerund.
how should we regard the syntactic function/grammatical relations of the word no in relation to the phrases it occurs in.