Remember that punctuation is a matter of style. The "rules" are those recognized by the various style manuals, and those vary between manuals, which themselves allow for exceptions and the judgment of the author. Which isn't to say that there aren't some universally recognized directives (like putting a period at the end of a complete sentence).
If you're writing under the direction of others -- for example, an editor or a professor -- you will likely have a style manual imposed on you. If you're writing for yourself, experimenting with an idiosyncratic style, then you may choose your punctuation to suit yourself. Just be aware of what other authors have done and what confusion may arise from the punctuation of more "standard" text.
James Joyce disdained all punctuation in the 4,391 words of Molly Bloom's soliloquy at the end of Ulysses. That's a series of remembrances and reflections of a character, but it attempts to replicate the stream of consciousness of someone on the verge of sleep, a sort of reverie.
The Chicago Manual of Style recommends dashes and ellipsis points to indicate interrupted speech. The latter is to be used to indicate the faltering speech of surprise and confusion:
Oh, Frank...I didn't see you...Why are you...? When did you....
The former to indicate a break in thought that's reflected in a break in the sentence structure:
The caller--and how great is the frustration of someone who's been on hold listening to on-hold music for an hour?--swore at the customer rep who finally came on the line.
If you use the google on "punctuating one-sided conversations" and "how do I write a telephone conversation", you'll find that ellipsis points are suggested if intervening descriptions are absent. Just remember to be careful with internet advice. You never know where it's been.