TestNG - re-run failed tests

Let’s assume you have several TestNG tests in a test class.

Part of them are very stable but others are not. 

public class RetryExampleTest {

    @Test
    public void stableTest() {
        ...
    }

    @Test
    public void unstableTest() {
        ...
    }

    @Test
    public void veryUnstableTest() {
        ...
    }
}

Sometimes or very often our unstable tests fail. We don’t want to re-run all our tests, only tests that failed.

IRetryAnalyzer

TestNG has the interface IRetryAnalyzer that contains only one method:

public interface IRetryAnalyzer {
    boolean retry(ITestResult testResult)
}

All you need is just to implement that method. If the method returns true it means the test should be re-run if failed.

Re-run failed tests 2 times example.

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer {

    private int retryCount = 0;
    private static int maxNumberOfRetriesForUnstableTests = 2;

    public boolean retry(ITestResult result) {
        retryCount++;
        if (retryCount <= maxNumberOfRetriesForUnstableTests) return true;
        return false;
    }

}

Different numbers of re-runs for different testes

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer {

    private int retryCount = 0;
    private static int maxNumberOfRetriesForUnstableTests = 2;
    private static int maxNumberOfRetriesForVeryUnstableTests = 3;

    public boolean retry(ITestResult result) {
        retryCount++;
        if (retryCount <= maxNumberOfRetriesForVeryUnstableTests 
                && result.getName().equals("veryUnstableTest")) return true;
        if (retryCount <= maxNumberOfRetriesForUnstableTests) return true;
        return false;
    }

}

Re-run failed tests n times if …

import org.joda.time.DateTime;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer {

    private int retryCount = 0;
    private static int maxNumberOfRetriesForUnstableTests = 2;
    private static int maxNumberOfRetriesForVeryUnstableTests = 3;
    private static int relaxDayMaxNumberOfRetries = 10;
    private static String rushDay = "Monday";
    private static String relaxDay = "Thursday";

    public boolean retry(ITestResult result) {
        retryCount++;
        if (isRushDayToday()) return false;
        if (isRelaxDayToday() && retryCount <= relaxDayMaxNumberOfRetries)  return true;
        if (retryCount <= maxNumberOfRetriesForVeryUnstableTests 
                && result.getName().equals("veryUnstableTest")) return true;
        if (retryCount <= maxNumberOfRetriesForUnstableTests) return true;
        return false;
    }

    private boolean isRushDayToday() {
        return new DateTime().dayOfWeek().getAsText().equals(rushDay);
    }

    private boolean isRelaxDayToday() {
        return new DateTime().dayOfWeek().getAsText().equals(relaxDay);
    }

}

Specify tests that should be re-run

import org.testng.annotations.Test;

public class RetryExampleTest {

    @Test
    public void stableTest() {
        // some test code
    }

    @Test(retryAnalyzer = RetryAnalyzer.class)
    public void unstableTest() {
        //  some test code
    }

    @Test(retryAnalyzer = RetryAnalyzer.class)
    public void veryUnstableTest() {
        //  some test code
    }

}

And the last - you can use several implementations of IRetryAnalyzer and apply different implementations for different tests.

Share how you implemented “re-run failed tests in TestNG” in the comments below or in How to re-run failed tests in TestNG using IRetryAnalyzer on GitHub.

You may also find these posts interesting: