Le hasard fait bien les tests

A presentation at DevFest Toulouse in November 2025 in Toulouse, France by David Pilato

Push code to production June 4th 1996 Original footage from ESA https://www.esa.int/ESA_Multimedia/Videos/1996/06/Ariane_501_recording_launch_pad_before_and_at_launch

public static int generateInt(int min, int max) { return ThreadLocalRandom.current().nextInt(min, max); } @Test public void testInteger() { int input = generateInt(Integer.MIN_VALUE, Integer.MAX_VALUE); int output = Math.abs(input); assertThat(output).isGreaterThanOrEqualTo(0); }

int output = Math.abs(input); assertThat(output).isGreaterThanOrEqualTo(0);

: https://github.com/dadoonet/fscrawler/actions/runs/14357866984/job/40251514398#step:4 296

Throwable #1: java.lang.NullPointerException: Cannot invoke “Elasticsearch.setIndex(String)” because the return value of “FsSettings.getElasticsearch()” is null

@Test public void settingsValidation() { FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … }

REPRODUCE WITH: mvn integration-test -Dtests.locale=az Throwable #1: java.lang.NullPointerException: Cannot invoke “Elasticsearch.setIndex(String)” because the return value of “FsSettings.getElasticsearch()” is null

@Test public void settingsValidationFr() { Locale.setDefault(new Locale.Builder().setLanguageTag(“fr”).build()); FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … } @Test public void settingsValidationAz() { Locale.setDefault(new Locale.Builder().setLanguageTag(“az”).build()); FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … } @Test public void settingsValidationEn() { Locale.setDefault(new Locale.Builder().setLanguageTag(“en”).build()); FsSettings settings = FsSettingsLoader.load();

Installation (pom.xml - Java) <dependency> <groupId>com.carrotsearch.randomizedtesting</groupId> <artifactId>randomizedtesting-runner</artifactId> <version>2.8.3</version> <scope>test</scope> </dependency>

Deactivate the default test plugin <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.19</version> <executions> <execution> <id>default-test</id> <phase>none</phase> </execution> </executions> </plugin>

Activate the randomized testing plugin <plugin> <groupId>com.carrotsearch.randomizedtesting</groupId> <artifactId>junit4-maven-plugin</artifactId> <version>2.8.3</version> … </plugin>

Configure the randomized testing plugin … <executions> <execution> <id>unit-tests</id> <phase>test</phase> <goals> <goal>junit4</goal> </goals> </execution> </executions>

Configure the randomized testing plugin … <configuration> <heartbeat>10</heartbeat> <jvmOutputAction>pipe,ignore</jvmOutputAction> <leaveTemporary>true</leaveTemporary> <ifNoTests>warn</ifNoTests> <listeners> <report-text showThrowable=”true” showStackTraces=”true” showOutput=”always” showStatusOk=”true” showStatusError=”true” showStatusFailure=”true” showStatusIgnored=”true” showSuiteSummary=”true” /> </listeners> <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> </systemProperties> </configuration> …

@Test public void testSeed() { Random generator = new Random(); int num = generator.nextInt(); assertThat(num).isEqualTo(1553932502); }

@Test public void testSeed() { Random generator = new Random(12345L); int num = generator.nextInt(); assertThat(num).isEqualTo(1553932502); }

… <configuration> <heartbeat>10</heartbeat> <jvmOutputAction>pipe,ignore</jvmOutputAction> <leaveTemporary>true</leaveTemporary> <ifNoTests>warn</ifNoTests> <listeners> <report-text showThrowable=”true” showStackTraces=”true” showOutput=”always” showStatusOk=”true” showStatusError=”true” showStatusFailure=”true” showStatusIgnored=”true” showSuiteSummary=”true” /> </listeners> <seed>${tests.seed}</seed> <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> </systemProperties> </configuration> …

REPRODUCE WITH: mvn integration-test -Dtests.seed=489581FE40712E4A # -Dtests.locale=az -Dtests.timezone=Asia/Ashkhabad

REPRODUCE WITH: mvn integration-test -Dtests.seed=489581FE40712E4A # -Dtests.locale=az -Dtests.timezone=Asia/Ashkhabad @Test @Seed(“489581FE40712E4A”) public void settingsValidationForLocaleAz() { // … }

@Test @Repeat(iterations = 10) public void repeatMe() { Locale locale = randomLocale(); DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofLocalizedDate(FormatStyle.FULL).withLocale(locale); String format = dateTimeFormatter.format(LocalDate.now()); System.out.println(“date is [” + format + “] with locale [” + locale.toLanguageTag() + “]”); }

<properties> <tests.locale>random</tests.locale> </properties> <configuration> … <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> <tests.locale>${tests.locale}</tests.locale> </systemProperties> </configuration>

private static final Locale savedLocale = Locale.getDefault(); @BeforeClass public static void setLocale() { String testLocale = System.getProperty(“tests.locale”, “random”); Locale locale = testLocale.equals(“random”) ? randomLocale() : new Locale.Builder().setLanguageTag(testLocale).build(); Locale.setDefault(locale); } @AfterClass public static void resetLocale() { Locale.setDefault(savedLocale); }

@Test public void withLocale() { DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofLocalizedDate(FormatStyle.FULL); String format = dateTimeFormatter.format(LocalDate.now()); System.out.println(“locale = ” + Locale.getDefault().toLanguageTag()); System.out.println(“format = ” + format); } # RUN 1 locale = es-PR format = miércoles 9 de marzo de 2016 # RUN 2 locale = tr-TR format = 09 Mart 2016 Çarşamba

mvn test -Dtests.locale=fr-FR 1> locale = fr-FR 1> format = mercredi 9 mars 2016

@Test public void sometimes() { int bulkSize = randomIntBetween(500, 1000); for (int i = 0; i < bulkSize; i++) { addDocument(“english_” + i, generatePerson()); if (frequently()) { addDocument(“french_” + i, generatePerson()); } if (rarely()) { // Shutdown Node 1 shutdownNode(1); } } }

@Test public void ignoreIfUseless() { boolean b = randomBoolean(); assertThat(b).isTrue(); }

@Test public void ignoreIfUseless() { boolean b = randomBoolean(); assumeTrue(b); assertThat(b).isTrue(); }

Do less during the day @Test @Nightly public void longRunningTest() { int bulkSize = randomIntBetween(500, 1000); for (int i = 0; i < bulkSize; i++) { try { Thread.sleep(Math.abs(between(100, 10000))); } catch (InterruptedException e) { assumeNoException(e); } } } $ mvn test IGNOR/A 0.00s | RandomTest.longRunningTest > Assumption #1: ‘nightly’ test group is disabled (@Nightly(value=))

Do more during the night <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> <tests.nightly>${tests.nightly}</tests.nightly> </systemProperties> $ mvn test -Dtests.nightly=true Started J0 PID(26908@MacBook-Pro-4.local). Suite: fr.pilato.demo.testframework.RandomTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:19, RandomTest.longRunningTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:29, RandomTest.longRunningTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:39, RandomTest.longRunningTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:49, RandomTest.longRunningTest OK 50.9s | RandomTest.longRunningTest stalled for 11.5s at: stalled for 21.5s at: stalled for 31.5s at: stalled for 41.5s at:

Do more during the night but stay reasonable @Test @Nightly @Timeout(millis = 10000) public void longRunningTest() { // … } $ mvn test -Dtests.nightly=true ERROR 10.0s | RandomTest.longRunningTest <<< > Throwable #1: java.lang.Exception: Test timeout exceeded (>= 10000 msec). > at __randomizedtesting.SeedInfo.seed([F4FC818C113EF9C6:A3FC90C16C6643D6]:0)

@RunWith(RandomizedRunner.class) public class RandomTest { @Test public void stopYourThreads() { new Thread(() -> { public void run() { while (true) { try { Thread.sleep(1000L); } catch (InterruptedException e) { } } } }, “zombie”).start(); } }

@RunWith(RandomizedRunner.class) @ThreadLeakFilters(filters = { FriendlyZombieFilter.class }) public class RandomTest { @Test public void identifyYourThreads() { new Thread(() -> { public void run() { while (true) { try { Thread.sleep(1000L); } catch (InterruptedException e) { } } } }, “friendly-zombie”).start(); } } public class FriendlyZombieFilter implements ThreadFilter { public boolean reject(Thread t) { return “friendly-zombie”.equals(t.getName()); } }

@Test(expected = NullPointerException.class) @Seed(“489581FE40712E4A”) public void testGestalt242() throws Exception { FsSettings settings = FsSettingsLoader.load(); // This is supposed to fail because of: // https://github.com/gestalt-config/gestalt/issues/242 settings.getElasticsearch().setIndex(getCurrentTestName()); fail(“The Gestalt project probably fixed issue 242”); }

David Pilato linkedin.com/in/dadoonet/ @dadoonet https://david.pilato.fr/

Le hasard fait bien les tests s de Sli https://speaker.pilato.fr/qC8sMm

David Pilato
@dadoonet

1 / 64

Le hasard fait bien les choses.

Si on applique cette idée aux tests unitaires ou aux tests d’intégration, on peut rendre nos tests beaucoup plus imprévisibles et du coup trouver des problèmes que notre esprit n’aurait jamais osé imaginer ! Par exemple, récemment, j’ai découvert dans une bibliothèque de gestion de configuration, un bug qui se produit lorsque la Locale est configuré en AZ. 🤦🏼‍♂️

Un autre exemple encore plus simple :

int input = generateInteger(Integer.MIN_VALUE, Integer.MAX_VALUE);
int output = Math.abs(input);

Peut générer -2147483648… Ce qui est assez inattendu pour une valeur absolue ! 😉 Les tests aléatoires peuvent découvrir ces cas tordus… C’est ce que l’équipe elasticsearch a mis en place depuis plusieurs années à l’aide du framework RandomizedTesting pour tester tout le code Java.

Après cette conférence, vous ne verrez plus jamais la fonction random() comme avant et découvrirez comment la (mal)chance peut vous aider ! 🍀

Resources

The following resources were mentioned during the presentation or are useful additional information.

DevFest Toulouse 2025

The conference website
Demo project

The source code for the demo project
RandomizedTesting framework

The framework provided by Carrot Search
Infinitest plugin for Intellij IDEA

The plugin to run tests in the background
Ariane 5 launch video

The Ariane 5 first launch on June 4th, 1996
FSCrawler project

FSCrawler is running tests with RandomizedTesting framework
A weird CI error in FSCrawler

The error I could not reproduce locally
Gestalt issue 242

The issue I opened after my findings

Le hasard fait bien les tests

Link for this presentation:

HTML code for embedding:

Share on social media:

Resources

DevFest Toulouse 2025

Demo project

RandomizedTesting framework

Infinitest plugin for Intellij IDEA

Ariane 5 launch video

FSCrawler project

A weird CI error in FSCrawler

Gestalt issue 242