Short answer: origin.
For usage "in the early days", it helps to know how the term was coined.
The OED cites a 1965 Communications [of the] ACM article as the first use of the term "source code".
Derek Jones, on The Shape of Code, references that OED entry and helpfully quotes C.ACM: "The PUFFT source language listing provides a cross-reference between the source code and the object code".
Derek also cites an earlier 1959 Proceedings of the EASTERN JOINT COMPUTER CONFERENCE: "The compiler uses this convention to aid in distinguishing between SIMCOM statements and SCAT instructions which may be included in the source code."
In the early days of computing, programming referred to the direct crafting of what we now call object code. Wikipedia's History of Programming Languages lists John Mauchly's Short Code of 1949 as the "one of the first high-level languages ever developed for an electronic computer". Although 1965 was about one and a half decades on from 1949, object code must have still loomed large when thinking about programming since even in the 1980s, people were still hand-crafting assembly code, which is very close to machine language / object code.
So with the object code as the primary reference, it makes sense to call the high-level language code its source. Today, the high-level code has become the primary reference, and it's common to replace the modifier source with the language name for newer languages (e.g. Java code). For older languages, you might see references to C source instead. The generic code tends to refer to the high-level language or source code these days.
This history leads me to conclude that the term source in source code carries the sense of origin - the origin of the object code.
The source/destination sense isn't all that far away from the origin sense, but source/destination would likely have implied having the source as the primary focus or both source and destination equally in view. I doubt that was the case in the early days.