16

I am trying to write some description of what I am currently fixing in my software tests (unit tests).

Basically, I have a situation where those tests work but they were not properly set up in a way I guarantee that they did work or that code they are testing is indeed correct.

They kind of worked by accident. If the test was properly set up some would probably fail.

So, I fixed them, making sure that someone doesn't inadvertently changes logic and tests are still passing but code is not correctly doing what is supposed to do anymore.

I know we use flaky for almost the opposite. Like when something unpredictably fails.

So, my question is: what is the appropriate term for when something unreliably works, like accidentally or by some sort of collateral effect?

CONTEXT

Context of usage is within software development.

When you have a test that is not reliable, because it fails from time to time, we call this test flaky.

What I wanted to know is a similar nomenclature for a test that looks correct and passes, but actually fails to test properly, meaning it is accidentally working.

SAMPLE SENTENCE

I am fixing this test because it was a fluke.

POSSIBLE ANSWERS

So far, I think I could use fluke, hit-and-miss or even fragile.

I have picked fluke because it seems to be concise enough and describe well the issue with having a test like that.

Improve this question
11
  • 1
    A "false negative" test? (I tested it and nothing went wrong, so it must be right.)
    – Ray Butterworth
    Apr 6, 2022 at 20:11
  • 1
    The two "setup"s are verbs, not nouns, so they should be "set up" (like "backup" and "back up").
    – Ray Butterworth
    Apr 6, 2022 at 20:13
  • 2
    Meanwhile, I've described this phenomenon in the past using the same word you use several times in your question: accidental. It "works only accidentally." You might use a word like "fortuitous" for a more "happy accident," but there's nothing happy about unstable code.
    – Andy Bonner
    Apr 6, 2022 at 20:52
  • 2
    @lucasarruda "fluke" sounds fine; "fluke one" doesn't sound right (I am a native speaker). "I am fixing this test because it was a fluke" should do fine.
    – Simon Crase
    Apr 8, 2022 at 8:45
  • 2
    For the sample sentence, I would suggest "I am fixing this test because it was it was faulty," or perhaps "I am fixing this test because it did not provide full coverage" would be accurate.
    – Andrew Morton
    Apr 8, 2022 at 13:26

16 Answers 16

Reset to default
27

It's possible to call it a fluke.

: a stroke of luck

The word is generally used to describe something good that happens but not due to merit. It happened due to some unexpected luck.

See example sentence:

Her second championship shows that the first one was no mere fluke.

Perhaps, you could say:

The code passing the test was only a fluke.

Improve this answer
7
  • 3
    I really like fluke. It see it does mean a stroke of luck which is about what I am searching for. Translated to Portuguese, it also means acaso, which basically is something that works by coincidence.
    – lucasarruda
    Apr 6, 2022 at 21:52
  • 4
    Fluke is an excellent choice, but IMO best suited for a one-time event, rather than unpredictable repeated behavior. Is that what you meant with your question?
    – DjinTonic
    Apr 7, 2022 at 23:15
  • @DjinTonic I think fluke is fine here. Deciding on the test was an unlucky 1-time event. Passing it repeatedly is a predictable consequence. We deal with these "1-time chance, then baked-in" situations all the time in coding. Saying 2 programs interleave in an arbitrary order means the 1st time is unknown, but the rest will probably be identical.
    – Owen Reynolds
    Apr 8, 2022 at 13:26
  • OK, I upvoted fluke.
    – DjinTonic
    Apr 8, 2022 at 13:28
  • I thought a fluke was a parasite.
    – user22542
    Apr 9, 2022 at 15:53
12

A couple of words come to mind:

  1. incidental or incidentally — happening by chance (source: Merriam-Webster).

    The test was not set up correctly, and only incidentally was passing.

  2. inadvertent or inadvertently, which has the same definition from Merriam-Webster (happening by chance).

    The test was not set up correctly, but was inadvertently passing.

    The test was inadvertently set up incorrectly, but was still passing.

Improve this answer
6
  • I really like incidental. Seems like a more technical alternative, to be used in a book or more formal discussion, then fluke. Though, in most (casual) situations, I would prefer the last one.
    – lucasarruda
    Apr 7, 2022 at 21:18
  • 1
    Reading it again, I think inadvertent is also a good technical choice, not only incidental. Those were really good suggestions, @greg-burghardt!
    – lucasarruda
    Apr 7, 2022 at 21:22
  • 1
    Note that the OP used inadvertent in the Q: "inadvertently changes logic". IMHO inadvertent refers to someone's direct mistaken act. The test doesn't inadvertently pass -- it passes because someone inadvertently forgot to [insert testing step].
    – Owen Reynolds
    Apr 8, 2022 at 13:32
  • @OwenReynolds: good point. I added a variation of your suggestion to my answer. Although I would still argue the test was inadvertently passing. From a developer's viewpoint, the "accidental" part could apply to the setup of the test or the outcome, in my opinion (both, frankly). I do like your suggestion, though.
    – Greg Burghardt
    Apr 8, 2022 at 14:10
  • The sense of MW’s definition 2 for incidental (“lacking importance”) comes across more strongly in this context.
    – Lawrence
    Apr 9, 2022 at 4:01
10

Your software functioned serendipitously. It was serendipity that it worked.

Dictionary
Serendipitous:
come upon or found by accident; fortuitous

Cambridge
Serendipity:
the fact of finding interesting or valuable things by chance

Improve this answer
8
  • 6
    I usually associate "serendipity" with an unexpected desirable outcome or event. A passing test that should be failing (or a test that always passes and never fails) is not a desirable outcome. The word "passing" in this context refers to the outcome of the test, but not necessarily the outcome the programmer intended or was expecting.
    – Greg Burghardt
    Apr 7, 2022 at 18:49
  • 1
    @GregBurghardt I can't find any reference to serendipity as an undesirable thing. Lexico, Collins, Cambridge, Merriam Webster, Dictionary, Britannica and others all associate it with good fortune and desirable things. Do you have a reference to the contrary?
    – Anton
    Apr 7, 2022 at 21:02
  • I think this could also fit, but I have picked fluke since I think it describes it better and also more concise than saying it's a "serendipitous test".
    – lucasarruda
    Apr 7, 2022 at 21:16
  • @lucasarruda You have chosen the best, and I agree that fluke is the best fit. I felt that my suggestion, although applicable and correct, is a little too literary and "high register" for your purposes.
    – Anton
    Apr 7, 2022 at 21:46
  • 2
    @GregBurghardt that's true. With serendipity it looks like the fact it's passing is a good thing, like it could not but lucky it does and that's good but, in reality, we would rather it had not because that tricked us into think the test was well done when it wasn't.
    – lucasarruda
    Apr 8, 2022 at 14:18
7

I've found "happens to work" is a useful phrase when writing about code (e.g. in Stack Overflow answers) in languages like C that have undefined behaviour. Code with UB is not guaranteed to break; that would require the compiler to make asm that specifically checks for that case. Instead, a compiler gets to assume that case doesn't happen.

If the resulting compiled executable happens to do what you want anyway, even though no language standard or documentation guarantees it, your code happens to work. (The critical implication is that if you called the same code from a different function that passed the same inputs, it might not work after a compiler inlined and optimized it in that other context.)

This is a very common problem with GNU C inline assembly, where you need to accurately describe to the compiler everything your code reads / writes. If you get this wrong it's undefined behaviour, but it's actually common for it to still work if you compile with optimization disabled. Or in a function that doesn't inline, because of calling-convention reasons. So it's very easy to write inline asm that happens to work, and writing tests to tell the difference requires some understanding of not only the documentation but also how compilers work and like to optimize.

GCC code that seems to break inline assembly rules but an expert believes otherwise on Stack Overflow is a case of someone believing that their code is correct (perhaps because tests passed), when in reality it only happens to work, and my answer demonstrates cases that break it.

That's an extreme case. Lots of things only guarantee behaviour when you meet the preconditions, but many of them break more easily or even verify those preconditions for you in a debug build.

I don't have usage citations other than my own writing (on SO such as the link above or this), but I believe the phrase is self-explanatory and fairly clearly captures the idea of being even less good than implementation-defined behaviour. (Ideally you can rely on portable language-standard behaviour, next best is code that is guaranteed by some implementations to work on them, and not acceptable is code that only happens to work on a given implementation as part of a given program. Unless you're playing code golf, then some people like to use ridiculous hacks.)

In a testing situation with a poorly designed test, e.g. with test cases where the right answer can be obtained for the right reason or the wrong reason, you could say your code "happens to pass". (Because of a fluke). But that's probably best limited to cases like that, not cases where you only tested some common easy cases and didn't detect failure broken corner cases. I haven't used "happens to pass" as a phrase before.

"Happens to work" may or may not apply to your case depending on the details, but is certainly useful for some cases covered by the question title.

Improve this answer
4
  • So, here we have someone who actually has personal experience of this. That's who gets my vote.
    – Lambie
    Apr 8, 2022 at 13:42
  • 1
    @peter-cordes happens to work explains this situation pretty well. What I wanted as kind of an adjective to describe this test, but this also a good alternative.
    – lucasarruda
    Apr 8, 2022 at 14:24
  • 1
    @lucasarruda Well, fluke ain't an adjective, fyi. :)
    – Lambie
    Apr 8, 2022 at 17:40
  • @Lambie you are correct I confused words. It's a noun. Sorry, ESL here.
    – lucasarruda
    Apr 8, 2022 at 20:10
4

term for when something unreliably works, like accidentally or by some sort of collateral effect?

hit or miss / hit and miss

hit or miss (phrase)

As likely to be unsuccessful as successful.

Her work can be hit-or-miss Lexico

hit and miss (adj.)

If something is hit and miss or hit or miss, it is sometimes successful and sometimes not. Collins

Until recently, software testing has been a hit-and-miss affair with few standards and even less consistency between programs. InfoWorld, Vol. 5, n.3 (1983)

The challenge of discovering issues with device and controller interaction without an automated solution is almost entirely hit or miss. D. Graham and M. Fewster; Experiences of Test Automation (2012)

Improve this answer
1
  • 1
    I like hit-and-miss. It seems to fit well and I think I could use that, though I am leaning a bit more towards fluke.
    – lucasarruda
    Apr 6, 2022 at 21:53
4

I would call it a "spurious test", or a "spurious pass", or say "the test passes spuriously". The test passes, but not in a correct way; the test passing does not really indicate what a passing test is supposed to indicate. (Wiktionary)

spurious (comparative more spurious, superlative most spurious)

False, not authentic, not genuine. His argument was spurious and had no validity.

A Google search for "spurious unit test" finds some examples of this usage, where a unit test is said to be "spurious" if it fails when the code it tests is actually correct, or passes when the code it tests is actually flawed, or the test result is otherwise incorrect with regard to the code under test.

Improve this answer
3

"Vacuous test" might fit, as in vacuous truth.

Informally, a logical statement is vacuously true if it is true but doesn't say anything; examples are statements of the form "everything with property A also has property B", where there is nothing with property A.

For example, if your tests check that "every known living animal on the Moon is a goat", the tests will happily pass.

What they should also check is that there's at least one living animal on the Moon.

Improve this answer
2

I've often tripped over the situation where software is simply presumed to work, but has never actually been tested. I term it having failed to fail

Improve this answer
1
  • Indeed, it failed to fail. That's also a good sentence to describe the problem.
    – lucasarruda
    Apr 10, 2022 at 18:41
2

Fellow software developer here, who also pays a lot of attention to tests.

When I talk to other developers about this exact situation, I use this phrase: "the test passes for the wrong reasons."

I know that's not a single word, but it's the plainest way I have found to express the fact that the test is written such that it fails to raise the alarm when the code-under-test misbehaves.

Improve this answer
1
  • That's who I described in the commit, but I wanted to know if a more succinct wording or even a single word to describe the issue, existed. I kind of thought about the idea of it existing because we call flaky a test that fails from time to time, intermittently. So I wondered if there would be a word that could describe this issue to most developers, like instantly when they hear it. But, yeah, definitely agree this is a good description for the explanation part of a commit (for the first sentence, I used fluke).
    – lucasarruda
    Apr 10, 2022 at 18:40
1

A term used in computer programming/engineering which seems to fit this situation reasonably well is kludge:

something, especially computer hardware or software, that has been put together from whatever is available, especially when it does not work very well.

From the Jargon File:

A crock that works. (A long-ago Datamation article by Jackson Granholme similarly said: “An ill-assorted collection of poorly matching parts, forming a distressing whole.”)

where a "crock" is defined as

A technique that works acceptably, but which is quite prone to failure if disturbed in the least.

Note: the Jargon File also has an entry for "kluge" (without the d), which has a related but noticably different meaning.

Improve this answer
1

How about this neologism: "fludgy" which gets the vibe of "fail", "flaky", "kludge", and "fluke". It'd be fun is that caught on: "That code is super fludgy" (floojee)

Improve this answer
2
  • 1
    Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Apr 9, 2022 at 16:10
  • Thanks for contributing! This site handles existing usage, though.
    – Andy Bonner
    Apr 13, 2022 at 21:33
0

Serendipitous is not exactly what you are looking for but if it "works" by accident then it could fit.

adjective: serendipitous
occurring or discovered by chance in a happy or beneficial way. "a serendipitous encounter"

Improve this answer
2
  • 1
    You should include a link to the source of your definition.
    – KillingTime
    Apr 8, 2022 at 18:24
  • This essentially duplicates an existing answer
    – Andy Bonner
    Apr 13, 2022 at 21:31
0

How about automagic (or automagical)? It is geek-talk for something that works, but is poorly understood and therefore functions mysteriously.

From dictionary.com:

adjective Digital Technology. (of a usually complicated technical or computer process) done, operating, or happening in a way that is hidden from or not understood by the user, and in that sense, apparently “magical”: I just downloaded an automagical update to my word processing software that somehow fixed the problems.

Improve this answer
0

How about "janky"?

jan·ky [ˈjaNGkē] adjective. North American. informal.

of extremely poor or unreliable quality: "the software is pretty janky"

You'll hear the word alot if you watch youtube channels related to computers.

Improve this answer
-1

I think the most appropriate word here is intermittent. As defined by Merriam-Webster, intermittent means coming and going at intervals. Whether a natural phenomenon or something occurring by design, something that happens occasionally is intermittent. Your program, then, would work intermittently.

Improve this answer
1
  • It wasn't intermittent, actually. It was working 100% of the times it ran. It was just doing so because it wasn't that well set up, preventing a wrong code from passing.
    – lucasarruda
    Apr 10, 2022 at 18:35
-1

Personally I would say that the test is either faulty, incorrect or inadequate because it does not actually verify that the unit under test will behave correctly in all conditions. As such it provides false reassurance (telling the tester that something has been tested passed when the test has not actually made sure that the code behaves correctly.

Possibly the best word to use is spurious - as in this is a spurious test but possibly may even be a superfluous test.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.