Is there any difference between "The letter was written by pencil" and "The letter was written in pencil" ?
There is no question that the usual expression is written in pencil, as shown by an Ngram that makes by pencil seem nonexistent:
Work executed with a pencil; graphite or a similar substance used as a medium for writing or drawing. Chiefly as in pencil.
1796 M. G. Lewis Monk II. iv. 33 A small note, addressed to me... It contained the following words, written in pencil. (OED online)
...I immediately after wrote down with pencil the substance of what he had said; this he afterwards amplified and corrected; these scraps of paper, written in pencil, I have either mislaid or destroyed, but recollect to have seen them since my arrival in England; ... Alexander Dalrymple; A reply to a "Letter from Andrew Stuart, Esq; to the Hon. the directors of the East-India Company" (1776)
With the use or help of; through: We came by the back road.
Through the agency or action of: was killed by a bullet. AHD
However by pen/pencil has been used, very infrequently, apparently when the author wanted to emphasize the means of writing (by means of or using a pen/pencil) over a description of the appearance of the writing, e.g., It's hard to tell from the photo, but it appears to be written in pencil.
Perhaps the better cases for using by in the following examples are: (1) "20 times faster than they can be written by pen or typewriter," where the by is clearly by means of. In doesn't sound right in this case for pencil, and we wouldn't say *written in typewriter. (2) "This is the first time this beautiful word [America] was ever written by pen or engraved by tool or stamped by type."
Note that written by pen has been used to mean simply authored or penned, without regard to the implement:
I call the Book of Job, apart from all theories about it, one of the grandest things ever written by pen. John Fritz; Theological Monthly, Vol. 2, p.77 (1922) [Setting aside the question of who authored it and how.]
Steinbeck used 300 pencils to write his novel East of Eden. Typewriters existed then, but Steinbeck preferred to write by pencil. Brianne Keith; American Lit 101 (2017)
Most of my poetry started out written by pen or pencil on whatever paper was available at the time, such as paper napkins (accompanying a coffee and the compulsory cookie or cake), or the inside of a emptied breakfast cereal box. Wayne Pendelton; The Closet Poet's Collection (2015)
For example, when a modern orchestra plays a Mozart symphony, a large number of detailed instructions have to be written by pencil into the score. Hungarian Music Guide, Issues 19-20, p.89 (1982) (Snippet view)
The question must be written by pencil (or at least by chemical pencil), legible, the words must be written completely without corrections, if it is possible. Edward Perdue; Lost Adventures (2005)
This is the first time this beautiful word [America] was ever written by pen or engraved by tool or stamped by type. John Thacher; The Continent of America (1971)
pencil footing a temporary total underneath a list of figures written by pencil. Guy Wanjialin; An International Dictionary of Accounting and Taxation (2004)
A will was disapproved where it was written by pencil and the scrivener and attesting witnesses were all persons benefitting from the purported will. US Dept. of the Interior; Digest of Indian Probate Law (1972)
The call number would have been written by pencil on the page following the title page and book pocket would have been pasted inside the book cover. Slade Gandert; Protecting Your Collection (1982)
However, the number of revisions in essays that were begun by pen and revised on the computer approached that in essays written by pen. Resources in Education, Vol. 24, Issues 1-3, p.46 (1989)
An indorsement written by pencil is sufficient, and it may be by a mark or with the figures 1-2-8 with no name, ... McKinney's Consolidated Law of New York Annotated (1991) (Snippet view)
The ADDRESSOGRAPH addresses and takes care of lists of names of every nature. It saves time and labor by ... —at the rate of 60 a minute— 20 times faster than they can be written by pen or typewriter.
Factory, Vol. 13, p.387 (1914)
The gelatin duplicators are used when 50 or less copies are needed. The material to be reproduced is either written by pen or typewriter with regular copying ink. It requires very little instruction to learn to operate this simple appliance. Frank McClelland; Office Training and Standards (1919)
Some verbs accept two or more prepositions more commonly:
In the country, people often make their own ink, and make it badly , the characters written by which are not so legible as those made by pencil. Pennsylvania State Reports (1869)
Some late alterations are then made in pencil again. David Clark; W.B. Yeats, the Writing of Sophocles' King Oedipus (1989)
When it is necessary to blend or smudge drawings made with pencil, charcoal, or pastel medium a stump is used. US Bureau of Navy Personnel; Illustrator Draftsman 3 & 2, p.55 (1972)