Randomized testing: Gotta Catch ‘Em All

@dadoonet @pilato.fr @david 14/05/2025 I’d like to cover today a framework we are using at Elastic in the Elasticsearch project. It’s in Java but actually the whole concept could be applied to any other language you want. I did not check. It might exist already.

One day, I was working on my FSCrawler project, you know, for files! And I got an error from Github actions where I’m running unit and integration tests anytime I’m pushing a new change as a Pull Request.

https://github.com/dadoonet/fscrawler/actions/runs/14357866984/job/40251514398#step:4:296 With that I can ensure the stability of the code when I’m changing the code base: well, it’s tests after all… And I like a lot TDD: Tests Driven Development So I got this error in a unit test.

Throwable #1: java.lang.NullPointerException: Cannot invoke “Elasticsearch.setIndex(String)” because the return value of “FsSettings.getElasticsearch()” is null @Test public void settingsValidation() { FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … } The error message is that you can not call “setIndex” on the Elasticsearch object here, because the Elasticsearch object actually does not exist. Although it should be loaded by FsSettingsLoader.load().

@Test public void settingsValidation() { FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … } So, as a developer, you try to run it locally from your IDE. And it does not fail. getElasticsearch() is not null. So why this is happening here? If look back at the maven failure report, here is what we can see..

REPRODUCE WITH: mvn integration-test -Dtests.locale=az Throwable #1: java.lang.NullPointerException: Cannot invoke “Elasticsearch.setIndex(String)” because the return value of “FsSettings.getElasticsearch()” is null When you look at the maven logs you can see this message which invites you to run a given command line. And if you run this command locally, then it fails indeed…

So the Locale here is something important. We do have a specific context where it fails vs where it does not. Should we write all the possible contexts? How many Locale do we have here? More than 1 thousand. How could we test that?

Welcome to Randomized Testing framework. It’s a framework built by Carrot Search. It’s an extension for JUnit and it alters the context of the execution of the tests, like the Locale, but also the Timezone and it will provide some random methods… Like:

So you have plenty of random. But what happens when you miss your target and your test fails? How can you reproduce the problem in the same context? You can of course log and print all the details of the Locale used, the Timezone, or whatever… How to make sure that you can run the same test locally and have the same “random” values?

A random number is actually never a random number. It’s always computed from a seed value which could be created from the timestamp for example. But if you provide a seed, the “random” values computed from that seed will always be the same, when executed in the same order. So running the test again with the given seed will create the exact same context.

REPRODUCE WITH: mvn integration-test -Dtests.seed=489581FE40712E4A -Dtests.locale=az -Dtests.timezone=Asia/Ashkhabad @Test @Seed(“489581FE40712E4A”) public void settingsValidationForLocaleAz() { // … } You can set the seed from the CLI or if you want to always test this context, you can create a new test and annotate it with the seed value. BTW you can also notice the timezone here which is different.

@Test @Repeat(iterations = 10) public void repeatMe() { Locale locale = randomLocale(); DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofLocalizedDate(FormatStyle.FULL).withLocale(locale); String format = dateTimeFormatter.format(LocalDate.now()); System.out.println(“date is [” + format + “] with locale [” + locale.toLanguageTag() + “]”); } Sometimes, you could repeat the same test multiple times within the same run. For that, you can use the Repeat annotation. It will run the same test 10 times but note the seed value here. It has a “subseed” value. Which is set from the first seed but for every new run of the same test.

@RunWith(RandomizedRunner.class) public class RandomTest { @Test public void stopYourThreads() { new Thread(new Runnable() { public void run() { while (true) { try { Thread.sleep(1000L); } catch (InterruptedException e) { } } } }, “friendly-zombie”).start(); } } Some other benefits of this framework, and this might not apply to other languages than Java, is that you can also detect for non terminated threads when you close the test suite. Normally, a Thread should be closed when you exit an application.

@RunWith(RandomizedRunner.class) public class RandomTest { @Test public void stopYourThreads() { new Thread(new Runnable() { public void run() { while (true) { try { Thread.sleep(1000L); } catch (InterruptedException e) { } } } }, “friendly-zombie”).start(); } } com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at fr.pilato.demo.testframework.RandomTest: 1) Thread[id=12, name=Thread-1, state=TIMED_WAITING, group=TGRP-RandomTest] at java.lang.Thread.sleep(Native Method) at fr.pilato.demo.testframework.RandomTest$1.run(RandomTest.java:180) at java.lang.Thread.run(Thread.java:745) at __randomizedtesting.SeedInfo.seed([1CD01D6C55CD93C0]:0) Here, it’s detected that you did not close the thread. And indeed, we just started a thread and never closed it explicitly so the JVM will keep it running forever until the JVM closes.

@RunWith(RandomizedRunner.class) @ThreadLeakFilters(filters = { FriendlyZombieFilter.class }) public class RandomTest extends RandomizedTest { @Test public void identifyYourThreads() { new Thread(new Runnable() { public void run() { while (true) { try { Thread.sleep(1000L); } catch (InterruptedException e) { } } } }, “friendly-zombie”).start(); } } public class FriendlyZombieFilter implements ThreadFilter { public boolean reject(Thread t) { if (“friendly-zombie”.equals(t.getName())) { return true; } return false; } } But sometimes you don’t have access to the code which is running. For example when you use a 3rd party library which is not coded the right way. May be you want to ignore that specific thread. The example shown here adds a filter which ignores a remaining thread named “friendly-zombie”.

https://github.com/gestalt-config/gestalt/issues/242 What happened to the failing test? I created a full reproduction script, outside the context of FSCrawler, without RandomizedTesting but just with the Locale set to AZ. The second test passes when using FR as a Locale but not the first one with AZ.

https://github.com/gestalt-config/gestalt/issues/242 It happened to be a problem when they are running a regular expression… They did not set the Locale properly which had some side effects here. After the PR was merged, I was able to confirm the behavior by running the test with the initial seed again. And it passed.

So let the tests run continuously with different contexts (different seeds) like a Roomba vacuum cleaner would do and let it catch all the errors for you.

So having your CI failing frequently is good and what you would like to happen. That means that after thousand of runs or more, you might have tested a lot of your code base.