Randomized testing: Gotta Catch ‘Em All

A presentation at Testμ Conf’25 by David Pilato

https://plugins.jetbrains.com/plugin/3146-infinitest

Push code to production ■ June 4th 1996 ■ Original footage from ESA https://www.esa.int/ESA_Multimedia/Videos/1996/06/Ariane_501_recording_launch_pad_before_and_at_launch

public static int generateInteger(int min, int max) { return ThreadLocalRandom.current().nextInt(min, max); } @Test public void testInteger() { int input = generateInteger( Integer.MIN_VALUE, Integer.MAX_VALUE); int output = Math.abs(input); assertThat(output, greaterThanOrEqualTo(0)); }

int output = Math.abs(input); assertThat(output, greaterThanOrEqualTo(0));

About Me David Pilato Developer | Evangelist linkedin.com/in/dadoonet/ @dadoonet https://david.pilato.fr/

https://github.com/dadoonet/fscrawler/actions/runs/14357866984/job/ 40251514398#step:4:296

Throwable #1: java.lang.NullPointerException: Cannot invoke “Elasticsearch.setIndex(String)” because the return value of “FsSettings.getElasticsearch()” is null

@Test public void settingsValidation() { FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … }

REPRODUCE WITH: mvn integration-test -Dtests.locale=az Throwable #1: java.lang.NullPointerException: Cannot invoke “Elasticsearch.setIndex(String)” because the return value of “FsSettings.getElasticsearch()” is null

@Test public void settingsValidationFr() { Locale.setDefault(new Locale.Builder().setLanguageTag(“fr”).build()); FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … } @Test public void settingsValidationAz() { Locale.setDefault(new Locale.Builder().setLanguageTag(“az”).build()); FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … } @Test public void settingsValidationEn() { Locale.setDefault(new Locale.Builder().setLanguageTag(“en”).build()); FsSettings settings = FsSettingsLoader.load();

Installation (pom.xml - Java) <dependency> <groupId>com.carrotsearch.randomizedtesting</groupId> <artifactId>randomizedtesting-runner</artifactId> <version>2.8.3</version> <scope>test</scope> </dependency>

Deactivate the default test plugin <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.19</version> <executions> <execution> <id>default-test</id> <phase>none</phase> </execution> </executions> </plugin>

Activate the randomized testing plugin <plugin> <groupId>com.carrotsearch.randomizedtesting</groupId> <artifactId>junit4-maven-plugin</artifactId> <version>2.8.3</version> … </plugin>

Configure the randomized testing plugin … <executions> <execution> <id>unit-tests</id> <phase>test</phase> <goals> <goal>junit4</goal> </goals> </execution> </executions>

Configure the randomized testing plugin … <configuration> <heartbeat>10</heartbeat> <jvmOutputAction>pipe,ignore</jvmOutputAction> <leaveTemporary>true</leaveTemporary> <ifNoTests>warn</ifNoTests> <listeners> <report-text showThrowable=”true” showStackTraces=”true” showOutput=”always” showStatusOk=”true” showStatusError=”true” showStatusFailure=”true” showStatusIgnored=”true” showSuiteSummary=”true” /> </listeners> <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> </systemProperties> </configuration> …

04 Random Spoiler: it’s not really random

@Test public void testSeed() { Random generator = new Random(); int num = generator.nextInt(); assertThat(num).isEqualTo(1553932502); }

@Test public void testSeed() { Random generator = new Random(12345L); int num = generator.nextInt(); assertThat(num).isEqualTo(1553932502); }

… <configuration> <heartbeat>10</heartbeat> <jvmOutputAction>pipe,ignore</jvmOutputAction> <leaveTemporary>true</leaveTemporary> <ifNoTests>warn</ifNoTests> <listeners> <report-text showThrowable=”true” showStackTraces=”true” showOutput=”always” showStatusOk=”true” showStatusError=”true” showStatusFailure=”true” showStatusIgnored=”true” showSuiteSummary=”true” /> </listeners> <seed>${tests.seed}</seed> <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> </systemProperties> </configuration> …

REPRODUCE WITH: mvn integration-test -Dtests.seed=489581FE40712E4A # -Dtests.locale=az -Dtests.timezone=Asia/Ashkhabad

REPRODUCE WITH: mvn integration-test -Dtests.seed=489581FE40712E4A # -Dtests.locale=az -Dtests.timezone=Asia/Ashkhabad @Test @Seed(“489581FE40712E4A”) public void settingsValidationForLocaleAz() { // … }

@Test @Repeat(iterations = 10) public void repeatMe() { Locale locale = randomLocale(); DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofLocalizedDate(FormatStyle.FULL).withLocale(locale); String format = dateTimeFormatter.format(LocalDate.now()); System.out.println(“date is [” + format + “] with locale [” + locale.toLanguageTag() + “]”); }

<properties> <tests.locale>random</tests.locale> </properties> <configuration> … <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> <tests.locale>${tests.locale}</tests.locale> </systemProperties> </configuration>

private static final Locale savedLocale = Locale.getDefault(); @BeforeClass public static void setLocale() { String testLocale = System.getProperty(“tests.locale”, “random”); Locale locale = testLocale.equals(“random”) ? randomLocale() : new Locale.Builder().setLanguageTag(testLocale).build(); Locale.setDefault(locale); } @AfterClass public static void resetLocale() { Locale.setDefault(savedLocale); }

@Test public void withLocale() { DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofLocalizedDate(FormatStyle.FULL); String format = dateTimeFormatter.format(LocalDate.now()); System.out.println(“locale = ” + Locale.getDefault().toLanguageTag()); System.out.println(“format = ” + format); } # RUN 1 locale = es-PR format = miércoles 9 de marzo de 2016 # RUN 2 locale = tr-TR format = 09 Mart 2016 Çarşamba

mvn test -Dtests.locale=fr-FR 1> locale = fr-FR 1> format = mercredi 9 mars 2016

@Test public void sometimes() { int bulkSize = randomIntBetween(500, 1000); for (int i = 0; i < bulkSize; i++) { addDocument(“english_” + i, generatePerson()); if (frequently()) { addDocument(“french_” + i, generatePerson()); } if (rarely()) { // Shutdown Node 1 shutdownNode(1); } } }

@Test public void ignoreIfUseless() { boolean b = randomBoolean(); assertThat(b).isTrue(); }

@Test public void ignoreIfUseless() { boolean b = randomBoolean(); assumeTrue(b); assertThat(b).isTrue(); }

Do less during the day @Test @Nightly public void longRunningTest() { int bulkSize = randomIntBetween(500, 1000); for (int i = 0; i < bulkSize; i++) { try { Thread.sleep(Math.abs(between(100, 10000))); } catch (InterruptedException e) { assumeNoException(e); } } } $ mvn test IGNOR/A 0.00s | RandomTest.longRunningTest > Assumption #1: ‘nightly’ test group is disabled (@Nightly(value=))

Do more during the night <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> <tests.nightly>${tests.nightly}</tests.nightly> </systemProperties> $ mvn test -Dtests.nightly=true Started J0 PID(26908@MacBook-Pro-4.local). Suite: fr.pilato.demo.testframework.RandomTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:19, RandomTest.longRunningTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:29, RandomTest.longRunningTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:39, RandomTest.longRunningTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:49, RandomTest.longRunningTest OK 50.9s | RandomTest.longRunningTest stalled for 11.5s at: stalled for 21.5s at: stalled for 31.5s at: stalled for 41.5s at:

Do more during the night but stay reasonable @Test @Nightly @Timeout(millis = 10000) public void longRunningTest() { // … } $ mvn test -Dtests.nightly=true ERROR 10.0s | RandomTest.longRunningTest <<< > Throwable #1: java.lang.Exception: Test timeout exceeded (>= 10000 msec). > at __randomizedtesting.SeedInfo.seed([F4FC818C113EF9C6:A3FC90C16C6643D6]:0)

@RunWith(RandomizedRunner.class) public class RandomTest { @Test public void stopYourThreads() { new Thread(() -> { public void run() { while (true) { try { Thread.sleep(1000L); } catch (InterruptedException e) { } } } }, “zombie”).start(); } }

@RunWith(RandomizedRunner.class) @ThreadLeakFilters(filters = { FriendlyZombieFilter.class }) public class RandomTest { @Test public void identifyYourThreads() { new Thread(() -> { public void run() { while (true) { try { Thread.sleep(1000L); } catch (InterruptedException e) { } } } }, “friendly-zombie”).start(); } } public class FriendlyZombieFilter implements ThreadFilter { public boolean reject(Thread t) { return “friendly-zombie”.equals(t.getName()); } }

https://github.com/gestalt-config/gestalt/issues/242

@Test(expected = NullPointerException.class) @Seed(“489581FE40712E4A”) public void testGestalt242() throws Exception { FsSettings settings = FsSettingsLoader.load(); // This is supposed to fail because of: // https://github.com/gestalt-config/gestalt/issues/242 settings.getElasticsearch().setIndex(getCurrentTestName()); fail(“The Gestalt project probably fixed issue 242”); }

Thanks! s e lid Do you have any questions? S David Pilato linkedin.com/in/dadoonet/ https://speaker.pilato.fr/6DyIt6

David Pilato
@dadoonet

1 / 64

Chance does things well.

If we apply this idea to unit tests or integration tests, we can make our tests much more unpredictable — and as a result, uncover issues that our minds would never have dared to imagine! For example, I recently discovered a bug in a configuration management library that occurs when the Locale is set to AZ. 🤦🏼‍♂️

Another, even simpler, example:

int input = generateInteger(Integer.MIN_VALUE, Integer.MAX_VALUE);
int output = Math.abs(input);

This can generate -2147483648… which is quite unexpected for an absolute value! 😉
Randomized tests can uncover these twisted edge cases… That’s what the Elasticsearch team has been doing for years using the RandomizedTesting framework to test all their Java code.

Add to that real integration tests using TestContainers, and you’ll have a complete approach to tests that regularly fail!

After this talk, you’ll never look at the random() function the same way again — and you’ll discover how (bad) luck can actually help you! 🍀

Resources

The following resources were mentioned during the presentation or are useful additional information.

TestMu Conference 2025

The conference website
Demo project

The source code for the demo project
RandomizedTesting framework

The framework provided by Carrot Search
A weird CI error in FSCrawler

The error I could not reproduce locally
Gestalt issue 242

The issue I opened after my findings

Buzz and feedback

Here’s what was said about this presentation on social media.

Randomized testing: Gotta Catch ‘Em All

Link for this presentation:

HTML code for embedding:

Share on social media:

Resources

TestMu Conference 2025

Demo project

RandomizedTesting framework

A weird CI error in FSCrawler

Gestalt issue 242

Buzz and feedback