"Unclear" is a reasonable word to use. You might also consider "opaque" if you want to communicate not only that it's unclear to you, but that it's generally unclear, or you could use "uncertain" if it's strictly unknown how the algorithm will behave.
However I think I would suggest reformulating your sentence to something closer to your description of what you want to say rather than looking for a particular single word choice. "The search behaviour of the algorithm is not clear to us" is fine, but you could also consider something like "We don't know what the search behaviour of this algorithm is", or "It is not clear to me how this search algorithm will behave".
If you are particularly interested in certain aspects of the behaviour of the algorithm, i.e. performance, optimality, or stability, you might be better to highlight the particular area that is unclear.