Using Test Listener to monitor status of Test Cases in a Test Suite

[robert.lee]

I have a Test Suite, which represents a Sanity Test Plan. The Test Suite is composed of several independent Test Cases. In the profile, there is a variable called “Test_is_ok”, which starts as true. As each Test Case is executed, the given Test Case may throw an exception, make Asserts or call markFailed.

In addition, a Test Listener is used to:

1. In the AfterTestCase hook, the following groovy code checks test case status and sets the global variable accordingly:
   GlobalVariable.Test_is_ok = testCaseContext.getTestCaseStatus() == ‘PASSED’
2. In the BeforeTestCase hook, the global variable Test_is_ok is checked, and ideally, the desired behavior is to skip all the remaining tests if an issue is detected:
   if (!GlobalVariable.Test_is_ok) {
   testCaseContext.skipThisTestCase()
   }

This looks good on paper, and yes, it does work, most of the time / some times. What I have found is that in between the AfterTestCase and the BeforeTestCase, for some unknown reason, the GlobalVariable.Test_is_ok goes from true to false. It seems to occur randomly. As such I have disabled the invocation to skipThisTestCase, and the entire Test Suite is always executed.

At this moment, I am just curious as to what is happening because I am perplexed as to how this could possibly happen.

Thanks in advance.

[system]

Hi there,

Thank you very much for your topic. Please note that it may take a little while before a member of our community or from Katalon team responds to you.

Thanks!

[kazurayam]

@robert.lee

In the BeforeTestCase hook, the global variable Test_is_ok is checked, and ideally, the desired behavior is to skip all the remaining tests if an issue is detected:
if (!GlobalVariable.Test_is_ok) {
testCaseContext.skipThisTestCase()
}

I think that this code never perfoms as you expect.

The skipThisTestCase() method of TestCaseContext object — what is this method designed for?

I think that this method is designed for Katalon Studio’s GUI where you configure a Test Suite. The GUI allows you to toggle a component Test Case is to be executed or skipped. See the following screenshot:

In the GUI, when you check off 🔲 Run of the Test Cases/TC1, the GUI will call skipThisTestCase() to mark the TC1 should be skipped.

A TestCaseContext must be configured before you start a Test Suite. Once the Test Suite TS1 has started, a call to testCaseContext.skipThisTestCase() in a Test Listener will have no effect. It’s too late.

[kazurayam]

Have a look at my previous experiment where I proposed an alternative architecture to make a Test Suite to finish as soon as possible once a comprising Test Case failed:

How to let TestCases in a TestSuite to quit once a TestCase failed

I have published a demo project on GitHub: Problem to solve A topic in the Katalon User Forum wrote: Stop all test suite if one test case failed Hi, there is a way to stop a test suite if one of its cases fails? I need some method or form to stop the suite, does anyone have a solution? Fair enough requirement, I think. Let me assume I have a Test Suite TS1 is comprised with 3 Test Cases: TC1 , TC2 , TC3 . The TC3 runs very long (e.g, 20 minutes). The TC2 normally passes but oc…

This way worked for me.