Perhaps the French term was picked up by English travellers visiting French speaking places (France and Switzerland), where the local language to describe their exchange offices. My guess is most of the tourists would change their money whilst abroad. The term then stuck and was by businesses used back at home, as the association of foreign-ness is appropriate.
Here's an Ngram of the terms in the English corpus. The capitalised and lowercase terms are roughly on par, and you can see the term becoming more popular.
And from the French corpus unsurprisingly has a larger scale, and starts earlier. Interesting to note that the capitalised term doesn't show up.
Checking some old books, bureau de change can be found in English 19th century travel guides for France and Switzerland, and also for money changers in London.
1858's The ABC, or, Alphabetical railway guide has adverts for S. J. Spry And Co. of 110 Leadenhill Street, London who are:
Money Changes, Foreign Coin and Bullion Merchants
Geld wechsel comptoir. Bureau de change.
The Welcome guest: Issues 37-75 of 1859 describes Switzerland:
So many large white buildings, too, with jalousied windows, on whose entablatured friezes you might read " Banque de Commerce," " Banque du Landgrafschaft," " Banque d'Escompte," " Banque et Bureau de Change."
A practical Swiss guide, by an Englishman in Switzerland of 1860 contains several Bureau de Change amongst its listings, such as:
Nouveautes, Silks, Shawls, &c. — Cluzeau Aine, opposite the Post.
Bureau de Change — Uytborck, 74 Montagne de la Cour : of the highest character.
Capital of Belgium — a smaller Paris. Manufact. lace. English Embassy : on the Royal Park.
The Bankers' magazine, and statistical register, Volume 19, Part 1 of 1864 translates some financial terms between English, French and Italian:
Exhange office - bureau de change, m. - casa de cambio, f..
Charles Dickens' 1885 journal Household Worlds appears to mention it several times, including these descriptions:
... a "Bureau de Change" stood temptingly before him ; English notes and coins displayed in profusion in its wire-protected window, affording ample evidence that a large business was done in the current coin of that realm.
Both letter and "At Home" were dated from the same house in Regent Street, but the forer bore the business heading, "Bureau de Change."