The “Cluedo” effect in randomised testing

When playing the board game Cluedo (also known as Clue in North America), you make a series of hypotheses that consist of a murderer, a weapon, and a room. In each hypothesis, the room has to be the room your piece is currently in, and the player to your left always has the first opportunity to invalidate your hypothesis by showing you one of their cards.

So, it’s quite infuriating when you discover that the player to your left owns the card of the room you’re in, because once they’ve shown you that card, you’re not going to get any more information — either from them or from any other player — until you’ve moved your piece out of that room. That’s because even if you change the murderer or the weapon in your hypothesis, they’re (presumably) just going to keep showing you their room card.

The situation is much the same with randomised testing, or fuzzing.

Once you’ve found one bug in the software-under-test, it’s often the case that you’ll keep finding that same bug again and again, even with different random test-cases, especially if the bug is in a commonly used feature. This phenomenon, which has been written about by Hughes et al. among others, can mask any other bugs that might be present. In this situation, the analogue to “moving your Cluedo piece out of the room” is either: (1) identifying the buggy feature and making sure that subsequent test-cases avoid it, or (2) fixing the bug in the software-under-test before proceeding.

Thanks to David MacIver for pointing me to Hughes et al.’s paper.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s