I am writing my PhD thesis, and I want to use the acronym "DOM" referring to "Domain Optimization Method", as I use a lot that expression throughout the document. At the moment of defining it (i.e. the first time I use it), I write something like: "...solved by using a Domian Optimization Method (DOM)..." My question is if here I should write "Domain", "Optimization" and "Method" starting with a capital letter.
No, unless all the words making up the initialism are proper nouns. In your case you would have to see if the words in your initialism represent proper nouns. I would advise you to take under advisement the comments under this answer, but for a more definitive answer I suggest looking it up in appropriate literature (i.e. in your university's library). Wikipedia does have a page on proper nouns, but I don't think it is very clear.
According to the AMA manual of style:
"Do not capitalize the words from which an acronym or initialism is derived (see, Abbreviations).prostate-specific antigen (PSA) enzyme-linked immunosorbent assay (ELISA) Exception: When the words that form the acronym or initialism are proper names, use capitals as described in [...]" 1
By the way, if you are in university, search your library for "style guide", it will probably have one or more books in it on style and this will most certainly be covered, probably under a heading like initialism(s) or (capitalisation).
1 American Medical Association. "Acronyms and Initialisms ." Acronyms and Initialisms - AMA Manual of Style. September 13, 2017. Accessed March 21, 2018. http://www.amamanualofstyle.com/view/10.1093/jama/9780195176339.001.0001/med-9780195176339-div1-113.
Subject to any specific guidance from your university's style guide, if it has one, using capitals in Domain Optimization Method at the point at which you define your abbreviation DOM is potentially helpful to the reader and appears to me to have no downside.
Are you intending to include an appendix giving a list of abbreviations or mathematical symbols in your thesis? If not, it would do no harm to redefine DOM in each chapter of the the thesis. As a reader of long scientific papers, I get very irritated if I am expected to keep in my mind some novel abbreviation defined many pages previously, so that if I have forgotten it I do not know where to look for the definition.