In Bengali if you want to say that somebody has a secret love affair with a woman, you say something that translates to drinking water on the sly. What is the proper English idiom for this?
You named it, do something on the sly.
on the sly: (Fig) secretly and deceptively: Martin was having an affair with the maid on the sly. (McGraw-Hill Dictionary of American Idioms and Phrasal Verbs)
Alternately, consider sneak around (with someone) and [do sonething] on the down-low (or DL).
sneak around: do something without someone's knowledge, especially to engage in romantic relationships: I suspect her husband has been sneaking around. I think his wife was sneaking around on him. (The American Heritage Dictionary of Phrasal Verbs)
You snuck around with him all summer and never told me. Not once. Out with some stupid, womanizing football player who would've broken your heart in two if given the chance. (Game for Trouble)
on the down-low: (AmEng, idiomatic, slang, euphemistic) Secretly sleeping with someone other than one's partner.Your Dictionary)
Reportedly, Eddie agreed to end the double dates with LeAnn, but that didn’t stop him from seeing her on the down-low. (Your Tango)
A bit on the side
a sexual relationship with someone who is not married to you, or the person you have the relationship with
An affair that’s being hidden is often one that is in some way or is for some reason forbidden, so in keeping with the “food and drink” category, “s/he/they is/are eating/tasting/enjoying the forbidden fruit” = “any pleasure or enjoyment regarded as illicit, esp sexual indulgence” (from The Free Dictionary).