I personally find it very hard to pronounce the 'L' sound right after 'N'. Would you say it is quite common or at least understandable to pronounce 'only' as 'own-knee' (fastly)?
To start with, what DanBron said. You are going to have problems if you leave it at that.
In my considerable experience of being a furriner with an accent, I find that it takes two to tango: that is, you have 1) the quality of the furriner's pronunciation and the effort zie puts into it, but also 2) the courtesy and the effort put into it by the native listener. This varies very widely. A linguist once told me that the British are tops at understanding their language when it is phonetically mistreated, by a process of extrapolation. Like when a Japanese in Switzerland asked me the way to "the rake". Some nations and individuals, however, will take no prisoners – they will resolutely refuse to understand a word if you are just ever so slightly off. In the same situation they are going to repeatedly tell the Japanese gentleman that there are no garden tools round here, and consider their duty done.
A similar predicament: for many years I suffered from the East Norwegian inability to understand a person with a non-rhotic or ulvular R. It is no good exhorting me to learn to trill my Rs, I just can't, the way I can't waggle my ears. The OP may empathise. I seriously thought about making a big badge with the letter R and tapping it, but moved to Bergen where they tend to a throaty R like a Parisian. Which I can do.
Maybe xiver77 can make a badge with the letter L? But seriously, some listeners are never, ever, going to make even the slightest effort to extrapolate from "ownee" or "ownree" to "only". Just avoid them if you can. They're the dummies not you.
According to the phonetician Jack Windsor Lewis, this pronunciation is commonly used by native speakers but can only occur when "only" comes before another word ("How English is Really Pronounced"). Also, as the other answers and comments on this page indicate, most native speakers don't think that they use this pronunciation. I certainly didn't.
An example of the many words which have weakforms which cause no stylistic problem is the word only. This is normal-sounding as given in EPD, LPD and ODP as containing an /l/ and it is invariably so pronounced in its prepausal occurrences but, whether stressed or not, on every other occasion it is overwhelmingly most often uttered throughout the whole educated native-English-speaking world without any /l/. There is little gain for the EFL learner to be given this information and the only admission of the existence of this majority form without /l/ is in LPD which quite wrongly categorises it as "non-RP". Another word very frequently to be heard without the /l/ with which it's spelt is certainly which is as often as not pronounced /`sɜːtn̩i/ or in a further reduction when not isolated as /`sɜːtni / with not even a syllabic /n/. Another such word is occasionally which is often /ə`keɪʒni/.
Note that he doesn't think it's useful for non-native speakers to know this information. I've just posted this answer because some people might be interested in knowing this.
With difficult consonant clusters, often the burden of pronunciation is pushed to the next syllable. A common way people say "only" is "oh-nlee". This is because there is a desire in English to reduce pauses between consonant sounds. (None of this is true for compound words, where emphasis often causes the word to still sound like two separate words: boatman, downhill, overboard)
You are probably less likely to be noticed to mispronounce the word "only" if in fact it is said as "oh-lee" than your suggested option (still spoken quickly, as you mentioned).