Cost = to require the payment of (in this context)
This book costs ten dollars. = This book requires the payment of ten dollars. The understanding of the exact meaning of the word 'cost' shall help us figure out its proper usage.
A question to enquire about its cost would begin with 'How much'. When a sentence begins with 'How much', an ESL student may proceed from the affirmative sentence, "This book costs ten dollars" and frame the question as follows:
"How much this book costs?".
But an interrogative sentence will have to be re-written as
"How much does this book cost?"
Why? Here's the rule: How much + auxiliary (do/does/did/will) + subject/pronoun + infinitive
Framing a question this way may be unique to English, which may pose a challenge to a foreign student.
Your other forms, both questions and answers, are therefore incorrect, grammatically.
However, "How much does it cost... this book?" may be okay in the informal/spoken language.