Briefly, what differences there are to tease out of the two terms boil down to these:
Treachery involves trickery; malicious deception. Something is presented as true and trusted and safe which in reality is secretly a fake that in some hidden way is willfully harmful.
Betrayals expose something that was supposed to be secure or secret in a way that brings unexpected harm. By extension, they can involve a deliberate breach of faith or confidence or allegiance, the breaking of one’s word or oath or promise. A modern way of expressing this is to double-cross someone or something.
Exploring the context
Because your two citations are from movies based on Tolkien’s written works, we can better understand how the author used those words by examining them in the context of his writings.
In The Lord of the Rings, words starting with “treacher-” occur 36 times, while those starting with “betray-” occur exactly half as often at 18 times. Incidentally, the figures for the related terms “traitor-” and “treason-” are 9 and 6 times respectively. Your first quote from the movie is taken directly from the book. It runs in full:
‘Quickly?’ said Treebeard. ‘Hoom! Yes, indeed. Quicker than I expected.
Indeed I have not seen them roused like this for many an age. We Ents do
not like being roused; and we never are roused unless it is clear to us
that our trees and our lives are in great danger. That has not happened in
this Forest since the wars of Sauron and the Men of the Sea. It is the
orc-work, the wanton hewing – rárum – without even the bad excuse of
feeding the fires, that has so angered us; and the treachery of a
neighbour, who should have helped us. Wizards ought to know better: they do
know better. There is no curse in Elvish, Entish, or the tongues of Men bad
enough for such treachery. Down with Saruman!’
Most mentions of treason and treachery involve Saruman, as the treason of Isengard is especially significant to the plot of the novel. He double-crossed both sides, secretly working against them both in pursuit of goals entirely his own.
Your second quote from the movies is not to be found in the books, at least not directly. Denethor never said that Théoden had betrayed Gondor, nor did he order the Guards of the Citadel to abandon their sworn posts.
But Tolkien does use betray- words often enough. Here’s a quote that includes both of the terms you are interested in which they are used in a contrasting way showing their distinct applications:
‘I cannot answer that now,’ said Gandalf. ‘Yet my heart guessed that Frodo
and Gollum would meet before the end. For good, or for evil. But of Cirith
Ungol I will not speak tonight. Treachery, treachery I fear;
treachery of that miserable creature. But so it must be. Let us
remember that a traitor may betray himself and do good that he does
not intend. It can be so, sometimes. Good night!’
He even later boils down that principle to the famous proverb:
Oft evil will
shall evil mar.
Betrayals in context
Here are other quotes involving betrayals, in no particular order:
‘This is grievous news concerning Saruman,’ he said; ‘for we trusted him
and he is deep in all our counsels. It is perilous to study too deeply the
arts of the Enemy, for good or for ill. But such falls and betrayals,
alas, have happened before. ...’
This was not at all to the liking of Gimli. ‘The agreement was made without
my consent,’ he said. ‘I will not walk blindfold, like a beggar or a
prisoner. And I am no spy. My folk have never had dealings with any of the
servants of the Enemy. Neither have we done harm to the Elves. I am no more
likely to betray you than Legolas, or any other of my companions.’
‘I did not mean the danger that we all share,’ said Frodo. ‘I mean a danger
to yourself alone. You swore a promise by what you call the Precious.
Remember that! It will hold you to it; but it will seek a way to twist it
to your own undoing. Already you are being twisted. You revealed yourself
to me just now, foolishly. Give it back to Sméagol you said. Do not say
that again! Do not let that thought grow in you! You will never get it
back. But the desire of it may betray you to a bitter end. You will
never get it back. In the last need, Sméagol, I should put on the Precious;
and the Precious mastered you long ago. If I, wearing it, were to command
you, you would obey, even if it were to leap from a precipice or to cast
yourself into the fire. And such would be my command. So have a care,
Sméagol!’
And as he waited, he felt, more urgent than ever before, the command that
he should put on the Ring. But great as the pressure was, he felt no
inclination now to yield to it. He knew that the Ring would only betray
him, and that he had not, even if he put it on, the power to face the
Morgul-king-not yet.
But it was no orc-chieftain or brigand that led the assault upon Gondor.
The darkness was breaking too soon, before the date that his Master had set
for it: fortune had betrayed him for the moment, and the world had
turned against him; victory was slipping from his grasp even as he
stretched out his hand to seize it. But his arm was long. He was still in
command, wielding great powers. King, Ringwraith, Lord of the Nazgûl, he
had many weapons. He left the Gate and vanished.
Not too soon came their aid to the Rohirrim; for fortune had turned against
Éomer, and his fury had betrayed him. The great wrath of his onset had
utterly overthrown the front of his enemies, and great wedges of his Riders
had passed clear through the ranks of the Southrons, discomfiting their
horsemen and riding their footmen to ruin. But wherever the mûmakil came
there the horses would not go, but blenched and swerved away; and the great
monsters were unfought, and stood like towers of defence, and the Haradrim
rallied about them.
In that hour of trial it was the love of his master that helped most to
hold him firm; but also deep down in him lived still unconquered his plain
hobbit-sense: he knew in the core of his heart that he was not large enough
to bear such a burden, even if such visions were not a mere cheat to
betray him. The one small garden of a free gardener was all his need
and due, not a garden swollen to a realm; his own hands to use, not the
hands of others to command.
Treachery in context
And here are a few quotes illustrating treacherous uses:
But after ages alone in the dark Gollum’s heart was black, and treachery
was in it. He slipped away, and returned to the island, of which Bilbo knew
nothing, not far off in the dark water.
‘A Ring of Power looks after itself, Frodo. It may slip off treacherously,
but its keeper never abandons it.
From Frodo’s mind the bright morning – treacherously bright, he thought –
had not banished the fear of pursuit; and he pondered the words of Gildor.
The marshes were bewildering and treacherous, and there was no
permanent trail even for Rangers to find through their shifting quagmires.
‘At first I feared, as Saruman no doubt intended, that Radagast had also
fallen. Yet I had caught no hint of anything wrong in his voice or in his
eye at our meeting. If I had, I should never have gone to Isengard, or I
should have gone more warily. So Saruman guessed, and he had concealed his
mind and deceived his messenger. It would have been useless in any case to
try and win over the honest Radagast to treachery. He sought me in good
faith, and so persuaded me.’
Paths were few and winding, and led them
often only to the edge of some sheer fall, or down into treacherous swamps.
Gimli strode forward undeterred, and
found that the water was shallow, no more than ankle-deep at the edge.
Behind him they walked in file, threading their way with care, for under
the weedy pools were sliding and greasy stones, and footing was treacherous.
Summary of Tolkien’s uses
So Tolkien often uses betray in the sense of giving something away that should not be, more often in fact than he uses it to mean the simpler sense of reversing one’s allegiance the way Peter Jackson did in the movie. Tolkien often uses treachery to mean hidden evil or perils, something actively and secretly malicious.
References
Tolkien knew his words. Indeed, he worked on the OED when he was younger. The OED says this of the verb betray. Remember that as an historical dictionary, the OED lists earlier senses first:
- To give up to, or place in the power of an enemy, by treachery or disloyalty.
- a. To be or prove false to (a trust or him who trusts one); to be disloyal to; to disappoint the hopes or expectations of.
b. figurative. To prove false to, let go weakly or basely.
- a. To lead astray or into error, as a false guide; to mislead, seduce, deceive (the trustful).
- To disclose or reveal with breach of faith (a secret, or that which should be kept secret).
- To reveal or disclose against one's will or intention the existence, identity, real character of (a person or thing desired to be kept secret).
Whereas treachery the OED says is:
- a. Deceit, cheating, perfidy; violation of faith or betrayal of trust; perfidious conduct.
And for treacherous:
- Of persons, their attributes or actions: Characterized by treachery; deceiving, perfidious, false; disloyal, traitorous.
- figurative. Of things: Deceptive, untrustworthy, unreliable; of ground, ice, etc., unstable, insecure.
Along with treason, Tolkien uses these words in many of those slightly different senses. But he never used perfidy or perfidious, which are a bit fancier, as infamously found in Perfidious Albion, a term first found in French. Over the centuries there were even more of these Latinate terms related to betrayal, treason, and treachery than remain current today, including the likes of bewray, betraise, traisement, trechetting, tradiment, and prodition.
Ironically, frankly all the words English now uses for these acts are Latinate terms acquired through French. Read into that what you will. Few Germanic terms like forsworn, oathbreaker, or trothless are used today even though they figured prominently in our most ancient literature. For example, Beowulf has the line which in the original Old English runs:
ac he sigewæpnum forsworen hæfde
Meaning that “he had forsworn victory-weapons”. (Albeit this instance is thought to be forsworn with the sense of bespelled.)
You might get away with forsworn these days, but I’d stay clear of trothless.
