Is there a word or phrase better or more common than "mentally expensive" to describe something that requires a lot of mental efforts? or "mentally expensive" or "mentally costly" are ok?
Example: It is a tedious task to understand and summarize this 1000-pages book; it is about theoretical mathematics and is written 700 years ago. It is a task that requires a lot of mental effort. It is mentally costly.
I am searching for a word or a phrase to replace what is written in italic.
In your particular context, you could use any of the words that imply hard work, e.g. an arduous task, or a laborious process, and this would be understood to mean in a mental sense. Tedious is also used to mean this sometimes (which you've already used in your example).
Alternatively, if you want to be explicit about the mental aspect of such a word, you could say it's mentally taxing, mentally consuming, mentally demanding or (as mentioned in the comments elsewhere) mentally draining
The term is cognitive load: total amount of mental effort being used.
A phrase from computer science is computationally expensive. One can think of the mental effort of understanding ancient mathematical theory as being a lot to compute. (And it's not too great a distance to remember when people held the profession of computer.)
I don't know if this is a big improvement over the OP's mentally expensive, as one would may need to explain computationally expensive to those other than computer scientists or mathematicians.
In other contexts, the OP might prefer some of the suggestions made in comments:
A superscription that has made its way into many theses and dissertations is:
Be warned, my son, of anything in addition to them. Of making many books there is no end, and much study wearies the body. (Ecclesiastes 12:12, NIV)
"It is a task that requires a lot of mental effort..." - It is a mentally strenuous task. - It is a task which bears a heavy burden on the mind. - It is a task that is taxing on the brain. - It is a task that requires great mental duress. - It is a task with a high cognitive load. - It is a mentally demanding task.
There are probably a few more suggestions I could throw out, but these are probably the ones that capture the essence of what you mean to say, though the meanings are not always 100% exact.
The choice of words here, I believe, depends on the tone (or the way) the sentence is conveyed. Namely, is it positive or negative? For example, are you having fun (e.g. you see it as a puzzle) trying to figure a 700 year old book, or do you only do it because you have to do it (e.g. you were assigned to) and you'd really rather do something else?
If the latter (negative) then most of the choices from previous answers would fit. Personally I would go with mentally exhausting cause I think that's the most common way to say it.
This 700 year old book is mentally exhausting!
If the former (positive) then I propose intellectually challenging (or mentally challenging)
(Warning: Do not confuse this with intellectually challenged which means the exact opposite!)
This 700 year old book is intellectually challenging!
Do note there is a fine line between the two because both exhausting and challenging imply difficulty of the task. Challenging however also appears to imply a reward for completing a task (in case of reading a math book it's the feeling of finally having grokked the material) while exhausting only focuses on its difficulty.
A word that would fit your example sentence would be brainpower:
brainpower
noun brain·pow·er \-ˌpau̇(-ə)r\
1 : the ability to think intelligently
It is a task that requires brainpower.
I can only think of slang to use after the activity. "My brain is fried after reading that 700 year old theoretical mathematics text."
