I've taken this explanation from here
Let's start by looking at so first:
So is used before an adjective for emphasis (to make the statement stronger).
Let’s compare these sentences:
so + adjective
The soup is hot.
The soup is so hot.
Let's now look at too:
John is too tall.
Too is used for emphasis also, but it means ‘more than needed’ or ‘more than enough.’
It is used to show that something is bad (negative) or that something is over the desired limit.
Take a look at these ideas:
John is so tall. He plays basketball well. (His tallness is not a bad thing)
John is too tall. He cannot sit comfortably on an airplane. (His tallness is a bad thing)
The second sentence means that he is very tall and that is a bad thing. His height is over the limit. He is 2 meters tall. Therefore he cannot fit on an airplane. John is too tall.
Following these rules, we still come across sentences that are similar in meaning. For example:
It’s so hot today. Let’s stay inside. (very hot)
It’s too hot today. Let’s stay inside. (there is too much heat so we cannot go outside)
Both of these sentences are correct and work well in this situation. In many situations though, one word is more suitable than the other. Look at this example:
The camera was ___ expensive, but I bought it.
The camera was ___ expensive. I didn’t buy it.
What’s the answer for #1?
So is the best answer. Why? If it were too expensive, then you couldn’t have bought it. It would be over the limit of the money you had (unless you had a credit card). The camera was so expensive (very expensive), but you bought it. The price was within your limits.
What’s the answer for #2?
Too. The price was too high. You didn’t have enough money. Therefore, you could not buy it. You could use ‘so’ here to mean it was very expensive. But if you want to express that the price was higher than you could afford, use too expensive.
In your examples
It's a nice book but it's 'too' expensive. I'm not going to buy it.
I would buy it if it weren't 'so' expensive."
you are expressing that
- you feel the price of the book is more than you can afford or want to pay for it.
- you would buy the book if you could afford it or if or it fitted your perception of value for money.