Grammatically speaking, your second sentence works. The comma is not only fine but normal.
However, the two sentences together do have a semantic issue. But I will explain that at the end of my answer.
Per Wheaton College:
When a sentence doesn’t begin with the subject but has instead an introductory word or phrase, a comma must separate the introduction from the rest of the sentence.
Despite his best efforts, the hero failed.
The comma goes between the introduction and the subject and must not separate the subject from the verb. Introductory elements often consist of prepositional phrases, subordinating conjunctions, participial phrases, or conjunctive adverbs.
It goes on to provide a list of words and phrases that can start a sentence in this fashion, all of which normally require a comma after them. In other words is one of those phrases.
Note, however, other sources of style and grammar dispute this absolute prescriptive advice.
For instance, The Chicago Manual of Style (17th ed.), 6.31, says:
Although an introductory adverbial phrase can usually be followed by a comma, it need not be unless misreading is likely. Shorter adverbial phrases are less likely to merit a comma than longer ones.
A comma should not be used to set off an adverbial phrase that introduces an inverted sentence.
Before the footlights stood one of the most notorious rakes of the twenty-first century.
But there is nothing wrong with the use of the comma in your sentence, and it's most likely how it would commonly be punctuated.
Using a different example, consider this:
"Two plus another number that is half of the first number's double equals five minus one."
"In other words, two plus two equals four."
There is also no reason that you can't have a sentence that is part statement and part question. In fact, this happens frequently—and commas are often used in such constructions.
You do love me, don't you?
If you were taller, do you think you would have been a basketball player?
He asked, "What about dinner?"
And going back to an earlier example:
What does two plus another number that is half of the first number's double equal? In other words, what does two plus two equal?
Which leads us back to your sentence:
✔ Do you think the process is complicated enough that users will need screenshots? In other words, could plain text instructions do the job?
Now, I said that the two sentences were grammatical but there was a semantic issue when putting them together.
Here's the problem:
The process is complicated enough that users will need screenshots.
(in other words)
Plain text instructions would do the job.
Literally speaking, the second sentence doesn't mean the same thing as the first sentence. In fact, it means the opposite.
If we use the phrase in other words (or something equivalent) we normally want to make sure that our simplified statement or question actually does mean the same thing as the more complicated statement or question that precedes it.
So, if you want to have your two sentences make complete sense when put next to each other, you need to reverse the logic of the second sentence:
Do you think the process is complicated enough that users will need screenshots? In other words, would plain text instructions be insufficient?