It's very unlikely for a planet to hit the Earth. And anyway, why is he so sure about it? He's not an astronomer."
It's very unlikely for a planet to hit the Earth. And by the way, why is he so sure about it? He's not an astronomer."
"And by the way, why is he..." has more results in Google Books than "and anyway, why is he..." Not sure if I should trust those figures though.
Should I use apple or pear in the sentence I love to drink apple / pear juice?
The answer is: it depends entirely on what message you want to convey.
By the way and anyway do not mean the same in that sentence:
It's very unlikely for a planet to hit the Earth. And anyway, why is he so sure about it? He's not an astronomer.
This version could roughly be rewritten as:
It's very unlikely for a planet to hit the Earth. And even if that was not so unlikely, there is another, maybe even more compelling argument: why is he so sure about it? He's not an astronomer.
Whereas the version with by the way would mean something like:
It's very unlikely for a planet to hit the Earth. And oh, yeah, there is another argument as well, let's mention that too: why is he so sure about it? He's not an astronomer.
Although the two expressions are sometimes used to indicate (more or less) the same meaning, I feel that anyway is stronger in introducing a compelling option, one that makes the previous point moot:
You say yes, I say no, but anyway, it's not our decision to make.
The fact that I say yes and you say no doesn't really matter.
It's very unlikely for a planet to hit the Earth. And anyway, why is he so sure about it? He's not an astronomer.
Is correct.
And, by the way, why is he so sure about it?
Is incorrect, because "By the way" implies you are introducing an unrelated subject. Cf. the comment above: "By the way, what's his wife's name?" where we have a departure from the previous argument, which would be a correct usage for "By the way".
These conjunctions do not have the same role:
English Text: System and structure Page 219 J.R. Martin - 1992
Conjunctions like incidentally, by the way, anyway and anyhow are more oriented to field than genre. They signal a change of topic, with
incidentally or by the way marking a departure from what has gone before and items like
anyway and anyhow marking a return.
_
Encyclopedic Graded Grammar Vol 2 - Page 644
Change of subject : By the way ; incidentally; right; all right; now
Dismissal of previous discourse : Anyway; anyhow; at any rate; at least; all the same;
_
Planning lessons and courses: designing sequences of work ... Tessa Woodward - 2004
Indicating change, e.g. in speaking, 'Oh, by the way ...', or in writing, 'An interesting digression here is ...' –
Suspension, e.g. in speaking, 'Just a sec', or continuation, e.g. 'Anyway, as I was saying . ...
_
The Syntax and Semantics of Discourse Markers - Page 42 Miriam Urgelles-Coll - 2010
Though both by the way and incidentally are classified by the authors as antonyms of anyway; all of them are markers of digression.
Knott and Dale classify anyway, in any case, and in any event as interruptions to return to a previous point.
The latter reference has a quite exhaustive discussion of anyway starting at this page.
