"Thinking out loud" in itself seems to imply literally speaking it with your voice, whereas I'm trying to describe "thinking out loud" silently by writing out thoughts on paper to work through a problem.
Brainstorming or freewriting is a term used by scholars and authors.
From the University of North Carolina at Chapel Hill:
When you freewrite, you let your thoughts flow as they will, putting pen to paper and writing down whatever comes into your mind.
Thinking out loud in English connotes that the person has not sufficiently formed the thought completely or thought the entire idea through, and so is merely processing the idea audibly without knowing its conclusion or validity. Doing something like this on paper (meaning writing it) might be referred to as "drafting" an idea or "sketching," or compiling notes. Additionally a "rough sketch" or a "rough draft" shows that the writing is not yet polished, edited, or completed.
Think it, jot it down TFD
to make a note of something
I keep reading your question and wanting to coin new phrases.
writing out loud
But that doesn't help if you're looking for an already existing word or phrase.
I've come across several ideas that express thinking aids (such as mind mapping and the already-mentioned brainstorming), but they aren't necessarily related to putting words on paper (or doing so on your own). Also, if you are "free associating," your writing will not be constrained, but if you are actually "working out a problem" (something specific) it's likely that what you write will have some purpose or framework behind it.
An alternative is to use a qualifier with a different term: a handwritten brain dump:
the act or an instance of comprehensively and uncritically expressing and recording one's thoughts and ideas (as on a particular topic)
I would tend to say "doodling" for this, although this could be interpreted as meaning truly random pictures or patterns rather than text or ideas, so some extra words might be needed; for instance "I'm just doodling on some ideas here, Bob" would probably have the right sense.
I might even use "noodling" in a similar way -- strictly this would describe "thinking out loud" with a musical instrument, but in the right context ("noodling with a pen and paper", "noodling on the keyboard") would probably convey the correct meaning.
You might like listing, according to this page from the Colorado State University website:
"Listing is a brainstorming technique many people find useful. It means doing just what its name suggests -- listing possible topics and then sublists of things you could say about each topic. A list could consist of the main topic of regional dialects and then sublists would be regional dialects you know or have experienced. Additional sublists might be particular words of each of those dialects, things you have noticed about those dialects (i.e. New Yorkers speak fast), what you think those dialects sound like, etc."
It doesn't focus on the writing part, but it doesn't have the connotation of involving sounds either.
Attribution: Writing@CSU. "Definition of Listing." The Writing Studio. Accessed May 13, 2018. https://writing.colostate.edu/guides/teaching/gentopic/pop4b.cfm.
In a review of a book Transforming Memories: Sharing Spontaneous Writing Using Loaded Words the Huffington Post writes
How does spontaneous writing work?
When we practiced spontaneous writing in our writing group, we would either put our finger on a random word in the dictionary, or with our eyes closed, select a photograph from a pile, or use some other technique to prompt our writing exercise. Then, we would just write about it for 10-20 minutes. Our intention was to be spontaneous—to start putting down our thoughts without any pre-determined intention. This approach has become very therapeutic when writing about challenging experiences and difficult memories.
I think that "putting your ideas to paper" is a standard phrase that's almost literally what you asked (replacing "thoughts" with "ideas").
There's also "putting your thoughts to paper" (search results skewed by "thoughts to paper" being a commercial name).
Journaling might fit, if the thinking is personal.
trans. To record in a journal.
1892 Idler May 461 His journaled impressions of America.
Forgive my non sequitur rambling here but wouldn't this simply be soliloquy on paper? Which is to say, if one were to ponder all the possible ideations of their mind, through keyboard or pen, it would nonetheless be monologue meant for audience and, ergo, a Shakespearian affect?
Forgive my soliloquy
Allow me to think out loud
Except that, classically, soliloquies were not always audible (to other characters) which, I think, might provide you the grey area you're looking for? I also very much agree with an earlier poster who suggested this as a prime opportunity to coin a new phrase...
I usually just warn that "this may be a bit non sequitur" and then ramble...