ATS : Test logging

Introduction


Good test reporting is crucial for serious automation testing. You need an easy way to:

  • Check which tests passed, which failed, and why
  • Analyze the performance of the tested application
  • Compare the results of running the same of different tests

Here we discuss different features of the ATS logging system


Flexible output

Here is where all logs can go

  • Test Explorer database
    this is a persisted database storage, the results are reviewed by Test Explorer web application. The project is located here
  • Console
    good when reviewing the results of one or a few tests
  • Mail report
  • custom

All log outputs are optional. You can use just one of them or some of them as you wish.

Based on log4j

The ATS logging system is currently configured to work along with log4j 1.2. As there are logging "bridges" like slf4j-log4j12 then this is not a big limitation.

Log4j is very flexible in terms of defining how much logging you need by specifying its log level. It is also possible to have more logs from some classes or some packages and less from others.
Note that currently Log4J 2.x is not supported but is planned for future releases.

Configure ATS Database Logging

Here are the steps to make it work for you.

Real-time, but non-blocking the test execution

The logging is performed in real-time. This means while your tests are running, you can look up the results in Test Explorer.

The test execution itself is not blocked waiting for the log event to be persisted in the database. The reason is that each log event is placed into a queue so the test execution can go on. While the test is running an additional dedicated thread moves these cached events to the database.

If you have very heavy logging, it may turn out it will take some time (some number of seconds) until the log event makes it to the database. So we have to say that real-time does not mean exactly immediately

Every participating party can log into the same testcase

Each Java application located anywhere on the test environment can be configured to send its log4j events into the same test case. This means for example that while doing some work with your Application Under Test(AUT), you can have log event from the Java API of the AUT into the same testcase.

Here is who can log:

  • The QA from the test
  • ATS libraries used in your tests
  • Actions running on some ATS agent
  • The Application Under Test
  • Third-party libraries

Test Explorer user guide


Test Explorer is a web application used to review the test results. It is a separate project from the ATS framework.

Here it is.

Set RUN information


Logging the JavaDoc


Specify custom log level on ATS agents


Increasing the speed we send some events to the log DB


Attach a file to the current testcase


Here we show how to place an important file next to your testcase data in Test Explorer

Printing Test Executor classpath on start


If you want to see(just once) in your console all the jars that are present in the Test Executor classpath, you have to set a system environment variable with the name "ats.log.classpath.on.start" and value "true".

For variable name you can use the AtsSystemProperties.LOG__CLASSPATH_ON_START constant.

Using Test Explorer database logging when running tests from non-java applications


Imagine you have your own tests written in a language different than java, but you would like to use Test Explorer for your needs.

What is not supported


Currently, we do not support simultaneous execution of more than one test case.

This is not affecting the performance testing as in that case a single test case causes the need load.



Back to parent page

Go to Table of Contents