(1) Both by doing X and in doing X form non-finite adjunct clauses of cause or reason. In the most general sense, they both mean 'as a consequence of doing X.' Consequently, there are many situations where both items, be and in, sound acceptable.
(1) [By / In refusing to ratify the Kyoto Protocol on global
warming], the administration may have set the stage for
(= As a consequence of refusing to ratify the Kyoto Protocol)
(2) However, the meaning of by doing X is wider, less restricted than the meaning of in doing X. When in doing X is acceptable, by doing X is generally also acceptable. The inverse is not true. There are specific cases where by doing X is fine, but in doing X is not.
Specifically, by doing X can not only denote any general reason (paraphrase: 'as a consequence of doing X') but is also used exclusively for direct causes - the action "doing X" is interpreted as an inherent, immediate cause, or typically the means, of a consequence. The action is a means to an end. The consequence is often brought about by the process of the action itself. (paraphrase for by doing X - 'as a DIRECT consequence of doing X').
(2a) [By cheating on his exams], he improved his grades.
(2b) ?* [In cheating on his exams], he improved his grades.
(= Here, "in" is odd because cheating is the direct cause, the means, the actual activity itself that leads to improved grades.)
In contrast, in doing X is only used to express indirect causes - the action "doing X" can always be potentially interpreted as the beginning of a causal chain with intermediate steps between the cause and its consequence. A process that follows from the action, not the action itself, is the cause. The consequence is often a side effect of the action (paraphrase for in doing X - 'as an INDIRECT consequence of doing X').
(3a) [In writing a public internet post], he opened himself up to ridicule.
(also acceptable: (3b) [By writing a public internet post], he opened himself up to ridicule.)
(= Writing is an indirect cause of ridicule. It's not the act of writing itself, but people reading the post, finding it silly, commenting on it etc., that causes the ridicule)
(4a) [In marrying him], she would satisfy her mother.
(also acceptable: (4b) [By marrying him], she would satisfy her mother.)
(= Not the marriage itself satisfies her mother, but the indirect consequences, being with a certain kind of man, his money, virtues etc.)
(5a) The government banned tobacco advertising and, [in doing so], contributed greatly to the nation's health.
(also acceptable: (5b) The government banned tobacco advertising and, [by doing so], contributed greatly to the nation's health.)
( = Not the ban itself, but indirect consequences, fewer people being exposed to tobacco ads etc., contribted to the nation's health)
(2a) It follows that, where only by and not in is acceptable, the non-finite clause can often be praphrased as by means of doing X. After all, one common type of a direct cause is a means.
(6a) the vulva is protruded [by (means of) lifting up the hips high]
(6b) ?* the vulva is protruded [in lifting up the hips high]
(= Lifting up the hips high is the direct cause of, the means to achieve, protraction of the vulva)
(7a) [By (means of) practising ten hours a day], you will become an expert.
(7b) ?* [In practising ten hours a day], you will become an expert.
(= The very act of practising itself will cause expertise, which makes "in" sound unnatural. (7b) could only be interpreted with the acquired expertise as an indirect, accidental or even undesired consequence of practising.)
(2b) As another consequence, by doing X often has the connotation of intention, deliberate causation. Thus, by doing X is often interpreted as 'because I (deliberately) did X.' Since in doing X cannot signify direct causation, the expression is bizarre in cases of intended consequences.
(8a) [By pulling out his gun], he (deliberately) wanted to scare the cashier.
(8b) ?* [In pulling out his gun], he (deliberately) wanted to scare the cashier.
(= Pulling a gun was intended to cause fear, so it's a direct consequence, a means. Hence, "in" is not acceptable here.)
(2c) A final consequence of the suggested semantic difference is that in doing X is often preferred where the consequence is unintentional, accidental, not beneficial. That's because the typical relation of in doing X is one of indirect consequences. (However, as always, by is also fine in these cases.)
(9) [In altering the rat's genes for longevity], the scientists accidentally also changed its intelligence.
(10) [In refusing to subscribe to one confession], I unfortunately received a lot of backlash from the religious communities.
(11) [In seeking to recover the stolen horse], he unintentionally stole another.
(= The focus on accidental, hence indirected, consequences work well with the "in doing X" structure.)
(3) The expression in doing X has been in decline for the last few hundred years. There is a good chance that it might eventually drop out of use completely. The following charts from COHA and Google Ngram illustrate:
Figure 1: Frequency per million words of "in V-ing" in COHA
Figure 2: Normalized frequency of "in V-ing" for five common verbs in Google Ngram
(3a) The most likely reason for the decline is the assumption that by doing X is more general than in doing X. In virtually all cases where the latter is acceptable, the former would also be acceptable. The inverse is not true. It thus makes sense to drop the distinction between direct and indirect consequences and retain only the more general construction that is permissible in all senses - the general meaning 'as a consequence of X' associated with by doing X.
(3b) The decline in in doing X structure leads, as is typical in such situations, to a change in its associated meanings. Specifically, in doing X is beginning to sound archaic, old-fashioned, formal, literary, stilted, posh, and upper-class. You can easily use the expression to sound intelligent, in scientific papers, in parliamentary speeches, in serious situations etc., but it feels inadequate to sound colloquial and modern, when you chat with your friends, when you write a facebook post, in casual situations etc.
(12a) Yeah, [by inviting this guy to my party], John really pissed me off!
(12b) (inappropriate:) # Yeah, [in inviting this guy to my party], John really pissed me off!
(= The formal connotations of "in doing X" clash with the informal character of the sentence.)
(13) The Government have been clear that, [in leaving the European Union], the UK will also leave its customs union.
(= The serious, political content of the sentence works well with the formal connotations of "in doing X")
(4) So, what does all of this mean for the initial example sentence?
- In principle both in doing this and by doing this are fine and the customer is unlikely to find either structure offensive.
- If the reference of "this" (in doing this) is the actual direct means, a resource, a gift, etc. that was given to the customer in order to reduce their frustration directly etc., then by is the best option. Can you paraphrase the sentence as by means of doing this?
- If the reference of "this" (in doing this) is supposed to express an indirect cause for the alleviation of the customer's troubles, perhaps an apology, store credit or bonus points that the customer may or may not use etc., then in is an elegant option. Are there intermediate steps between your action and the potential improvement in customer satisfaction?
- Do you want to stress your own effort? Do you want to focus on the energy you invested to get a direct response from the customer? Do you want to get credit for your action? Then go with by.
- Do you want to sound more apologetic? Do you want to show that it is up to the customer to accept your offer? Do you want to underline the indirect link between your action and the customer's power to forgive you? Then go with in.
- If you want to conduct your business relations on a formal, distant, serious, rigid level ("Dear Sir", "We look forward to hearing from you."), then in doing X is a great formulation.
- If you prefer a more colloquial, casual, informal, friendly way to talk to your customers ("Dear Bill", "Please let me know what you think about this"), then in doing X may come across as too snobbish, conceited or distant.