Test coverage

https://plugins.jetbrains.com/plugin/3146-infinitest

Push code to production ■ June 4th 1996 ■ Original footage from ESA https://www.esa.int/ESA_Multimedia/Videos/1996/06/Ariane_501_recording_launch_pad_before_and_at_launch

public static int generateInteger(int min, int max) { return ThreadLocalRandom.current().nextInt(min, max); } @Test public void testInteger() { int input = generateInteger( Integer.MIN_VALUE, Integer.MAX_VALUE); int output = Math.abs(input); assertThat(output, greaterThanOrEqualTo(0)); }

int output = Math.abs(input); assertThat(output, greaterThanOrEqualTo(0));

About Me David Pilato Developer | Evangelist linkedin.com/in/dadoonet/ @dadoonet https://david.pilato.fr/

01 The bug

github.com/dadoonet/fscrawler/

https://github.com/dadoonet/fscrawler/actions/runs/14357866984/job/ 40251514398#step:4:296

Throwable #1: java.lang.NullPointerException: Cannot invoke “Elasticsearch.setIndex(String)” because the return value of “FsSettings.getElasticsearch()” is null

Throwable #1: java.lang.NullPointerException: Cannot invoke “Elasticsearch.setIndex(String)” because the return value of “FsSettings.getElasticsearch()” is null @Test public void settingsValidation() { FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … }

@Test public void settingsValidation() { FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … }

REPRODUCE WITH: mvn integration-test -Dtests.locale=az Throwable #1: java.lang.NullPointerException: Cannot invoke “Elasticsearch.setIndex(String)” because the return value of “FsSettings.getElasticsearch()” is null

@Test public void settingsValidationFr() { Locale.setDefault(new Locale.Builder().setLanguageTag(“fr”).build()); FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … } @Test public void settingsValidationAz() { Locale.setDefault(new Locale.Builder().setLanguageTag(“az”).build()); FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … } @Test public void settingsValidationEn() { Locale.setDefault(new Locale.Builder().setLanguageTag(“en”).build()); FsSettings settings = FsSettingsLoader.load();

02 Installation

Installation (pom.xml - Java) <dependency> <groupId>com.carrotsearch.randomizedtesting</groupId> <artifactId>randomizedtesting-runner</artifactId> <version>2.8.3</version> <scope>test</scope> </dependency>

Deactivate the default test plugin <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.19</version> <executions> <execution> <id>default-test</id> <phase>none</phase> </execution> </executions> </plugin>

Activate the randomized testing plugin <plugin> <groupId>com.carrotsearch.randomizedtesting</groupId> <artifactId>junit4-maven-plugin</artifactId> <version>2.8.3</version> … </plugin>

Configure the randomized testing plugin … <executions> <execution> <id>unit-tests</id> <phase>test</phase> <goals> <goal>junit4</goal> </goals> </execution> </executions>

Configure the randomized testing plugin … <configuration> <heartbeat>10</heartbeat> <jvmOutputAction>pipe,ignore</jvmOutputAction> <leaveTemporary>true</leaveTemporary> <ifNoTests>warn</ifNoTests> <listeners> <report-text showThrowable=”true” showStackTraces=”true” showOutput=”always” showStatusOk=”true” showStatusError=”true” showStatusFailure=”true” showStatusIgnored=”true” showSuiteSummary=”true” /> </listeners> <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> </systemProperties> </configuration> …

03 Usage

04 Random Spoiler: it’s not really random

@Test public void testSeed() { Random generator = new Random(); int num = generator.nextInt(); assertThat(num).isEqualTo(1553932502); }

@Test public void testSeed() { Random generator = new Random(12345L); int num = generator.nextInt(); assertThat(num).isEqualTo(1553932502); }

… <configuration> <heartbeat>10</heartbeat> <jvmOutputAction>pipe,ignore</jvmOutputAction> <leaveTemporary>true</leaveTemporary> <ifNoTests>warn</ifNoTests> <listeners> <report-text showThrowable=”true” showStackTraces=”true” showOutput=”always” showStatusOk=”true” showStatusError=”true” showStatusFailure=”true” showStatusIgnored=”true” showSuiteSummary=”true” /> </listeners> <seed>${tests.seed}</seed> <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> </systemProperties> </configuration> …

REPRODUCE WITH: mvn integration-test -Dtests.seed=489581FE40712E4A # -Dtests.locale=az -Dtests.timezone=Asia/Ashkhabad

REPRODUCE WITH: mvn integration-test -Dtests.seed=489581FE40712E4A # -Dtests.locale=az -Dtests.timezone=Asia/Ashkhabad @Test @Seed(“489581FE40712E4A”) public void settingsValidationForLocaleAz() { // … }

@Test @Repeat(iterations = 10) public void repeatMe() { Locale locale = randomLocale(); DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofLocalizedDate(FormatStyle.FULL).withLocale(locale); String format = dateTimeFormatter.format(LocalDate.now()); System.out.println(“date is [” + format + “] with locale [” + locale.toLanguageTag() + “]”); }

<properties> <tests.locale>random</tests.locale> </properties> <configuration> … <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> <tests.locale>${tests.locale}</tests.locale> </systemProperties> </configuration>

private static final Locale savedLocale = Locale.getDefault(); @BeforeClass public static void setLocale() { String testLocale = System.getProperty(“tests.locale”, “random”); Locale locale = testLocale.equals(“random”) ? randomLocale() : new Locale.Builder().setLanguageTag(testLocale).build(); Locale.setDefault(locale); } @AfterClass public static void resetLocale() { Locale.setDefault(savedLocale); }

@Test public void withLocale() { DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofLocalizedDate(FormatStyle.FULL); String format = dateTimeFormatter.format(LocalDate.now()); System.out.println(“locale = ” + Locale.getDefault().toLanguageTag()); System.out.println(“format = ” + format); } # RUN 1 locale = es-PR format = miércoles 9 de marzo de 2016 # RUN 2 locale = tr-TR format = 09 Mart 2016 Çarşamba

mvn test -Dtests.locale=fr-FR 1> locale = fr-FR 1> format = mercredi 9 mars 2016

@Test public void sometimes() { int bulkSize = randomIntBetween(500, 1000); for (int i = 0; i < bulkSize; i++) { addDocument(“english_” + i, generatePerson()); if (frequently()) { addDocument(“french_” + i, generatePerson()); } if (rarely()) { // Shutdown Node 1 shutdownNode(1); } } }

05 But also

@Test public void ignoreIfUseless() { boolean b = randomBoolean(); assertThat(b).isTrue(); }

@Test public void ignoreIfUseless() { boolean b = randomBoolean(); assumeTrue(b); assertThat(b).isTrue(); }

@Test public void ignoreIfUseless() { boolean b = randomBoolean(); assumeTrue(b); assertThat(b).isTrue(); }

Do less during the day @Test @Nightly public void longRunningTest() { int bulkSize = randomIntBetween(500, 1000); for (int i = 0; i < bulkSize; i++) { try { Thread.sleep(Math.abs(between(100, 10000))); } catch (InterruptedException e) { assumeNoException(e); } } } $ mvn test IGNOR/A 0.00s | RandomTest.longRunningTest > Assumption #1: ‘nightly’ test group is disabled (@Nightly(value=))

Do more during the night <systemProperties combine.children=”append”> <arg.common>arg.common</arg.common> <tests.nightly>${tests.nightly}</tests.nightly> </systemProperties> $ mvn test -Dtests.nightly=true Started J0 PID(26908@MacBook-Pro-4.local). Suite: fr.pilato.demo.testframework.RandomTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:19, RandomTest.longRunningTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:29, RandomTest.longRunningTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:39, RandomTest.longRunningTest HEARTBEAT J0 PID(26908@MacBook-Pro-4.local): 2016-03-10T18:04:49, RandomTest.longRunningTest OK 50.9s | RandomTest.longRunningTest stalled for 11.5s at: stalled for 21.5s at: stalled for 31.5s at: stalled for 41.5s at:

Do more during the night but stay reasonable @Test @Nightly @Timeout(millis = 10000) public void longRunningTest() { // … } $ mvn test -Dtests.nightly=true ERROR 10.0s | RandomTest.longRunningTest <<< > Throwable #1: java.lang.Exception: Test timeout exceeded (>= 10000 msec). > at __randomizedtesting.SeedInfo.seed([F4FC818C113EF9C6:A3FC90C16C6643D6]:0)

@RunWith(RandomizedRunner.class) public class RandomTest { @Test public void stopYourThreads() { new Thread(() -> { public void run() { while (true) { try { Thread.sleep(1000L); } catch (InterruptedException e) { } } } }, “zombie”).start(); } }

@RunWith(RandomizedRunner.class) public class RandomTest { @Test public void stopYourThreads() { new Thread(() -> { public void run() { while (true) { try { Thread.sleep(1000L); } catch (InterruptedException e) { } } } }, “zombie”).start(); } } com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at fr.pilato.demo.testframework.RandomTest: 1) Thread[id=12, name=zombie, state=TIMED_WAITING, group=TGRP-RandomTest] at java.lang.Thread.sleep(Native Method) at fr.pilato.demo.testframework.RandomTest$1.run(RandomTest.java:180) at java.lang.Thread.run(Thread.java:745) at __randomizedtesting.SeedInfo.seed([1CD01D6C55CD93C0]:0)

@RunWith(RandomizedRunner.class) @ThreadLeakFilters(filters = { FriendlyZombieFilter.class }) public class RandomTest { @Test public void identifyYourThreads() { new Thread(() -> { public void run() { while (true) { try { Thread.sleep(1000L); } catch (InterruptedException e) { } } } }, “friendly-zombie”).start(); } } public class FriendlyZombieFilter implements ThreadFilter { public boolean reject(Thread t) { return “friendly-zombie”.equals(t.getName()); } }

Random execution order

06 Back to the bug

@Test public void settingsValidation() { FsSettings settings = FsSettingsLoader.load(); settings.getElasticsearch().setIndex(getCurrentTestName()); // … }

https://github.com/gestalt-config/gestalt/issues/242

https://github.com/gestalt-config/gestalt/issues/242

@Test(expected = NullPointerException.class) @Seed(“489581FE40712E4A”) public void testGestalt242() throws Exception { FsSettings settings = FsSettingsLoader.load(); // This is supposed to fail because of: // https://github.com/gestalt-config/gestalt/issues/242 settings.getElasticsearch().setIndex(getCurrentTestName()); fail(“The Gestalt project probably fixed issue 242”); }

07 Limitations

Thanks! s e lid Do you have any questions? S David Pilato linkedin.com/in/dadoonet/ https://speaker.pilato.fr/6DyIt6