Both are based on librsync; apart from that, they behave quite differently.
Both are based on librsync but beyond that they behave quite differently.
Both are based on librsync but the layers above that behave quite differently.
The 'above' in the original feels clumsy - wrong; I much prefer 'beyond' if only a single-word change is allowed. I prefer my first version to the other two. My second example, without any commas, is extreme in its (lack of) punctuation, but is probably tenable. With 'beyond' instead of 'above', options 1 and 3 in the question are OK. My third option uses 'above', but the extra context added by 'layers' makes it sensible.
Both are based on librsync but, beyond that, they behave quite differently.
I think this punctuation is also tenable (if you have any comma around 'beyond that' - or 'above that' in the original - then you need the pair since it is a parenthetical comment in the sentence and the pair of commas makes that clear). If you dropped the comma-enclosed phrase, you would not need any punctuation:
Both are based on librsync but they behave quite differently.
You could add a comma before 'but', and in times past that would have been added automatically.