I know that I can say "I was drawn to someone," but can I also say "I was drawn to something by X"? I want to imply how I came across a topic that later became a serious project for me. Or should I simply use "come across"?
"Come across" connotes a particular happenstance, usually without much meaning about one's internal state or reaction to it.
You can definitely use "drawn to" for a topic. One can imagine using them together, like
As a child, Mike Duncan came across Gibbon's books at a relative's house. As Duncan read them, he found that he was drawn to the study of Roman History.