Skip to main content
Skip table of contents

Running Regression Tests From Command Line

You can run regression test from command line. This is needed, if you want to e.g. integrate the regression tests into a continuous integration server such as Jenkins.

Call the Regression Test Command Line Tool like:

CODE
java -jar RegTestRunner.jar -project <path to the Builder project folder> 
[-suite <name of the test suite to be executed>] 
[-logfile <filename including path>] 
[<target Bridge>]

e.g. java -jar RegTestRunner.jar -project "C:\xUML Documentation\Advanced Modeling\PState" -suite "Build" -logfile C:\Temp\testlog.xml.

Click here to download the example file: Builder project Advanced Modeling/PState

<your example path>\Advanced Modeling\PState\uml\pstatePurchaseOrder.xml

Parameters

Regression Test Parameters

To select the tests to perform, you can specify the following parameters:

Parameter

Manda- tory

Description

Example

-project

(tick)

Specify the path to the Builder project that contains the regression tests.

-project "C:\E2E Documentation\Advanced Modeling\PState"

-suite

Specify the test suite you want to run. The test suite must exist within the project given with -project. If you specify no test suite, all tests of the project will be executed.

-suite "QA Tests/PurchaseOrderExample"

-file

Deprecated since Analyzer 6.0.27Specify the path to a testsuite.xml that contains the tests you want to perform. For more information on where to find such a file, refer to Regression Test File and Folder Structure.

-file "C:\E2E Documentation\Advanced Modeling\PState\regressiontest\QA Tests\testsuite.xml"

-list

Lists all available test suites in the project specified by -project. No tests executed.

-project "C:\E2E Documentation\Advanced Modeling\PState" -list

-logfile

Optionally specify a filename (including extension of your choice) and a path. The RegTestRunner will generate a file with the given name to the given path. If you specify a filename only, the logfile will be generated to the location of the test suite.

This logfile is not to be confused with the logfile that is created if you have added a logfile option in the Regression Test Tool (see Logging Test Runs).

For more information on the contents of the logfile, see Format of the Test Logfile further below.

-logfile C:\Temp\myLogFile.txt

Target Bridge Parameters

Optionally specify a target Bridge to run the tests against. Specify all or none of the following parameters:

Parameter

Description

Example

-host

Host name of the Bridge. Used for SOAP, deploy, start and stop tests.

-host testbridge.e2e.ch

-port

Bridge port. Used for deploy, start and stop tests.

-port 8080

-username

Bridge user to authenticate for deploy, start and stoptests.

-username admin

-password

Password of the Bridge user. Used for deploy, start and stop tests.

-password secure1234

-node

Bridge node name. Used for deploy, start and stop tests. If it is the same as -host, you can skip this parameter.

-node localhost

Note, that the target Bridge parameter values will only be applied to the tests, if the tests do not have these values specified directly in the test themselves. Tests that have e.g. specified a dedicated host in the test case properties will not get overwritten by the target Bridge parameter values.

To make your tests compliant to using the target Bridge parameters, you have to specify the related data (host, port, user credentials and node) via test case options. How to do that is explained in detail on Adding Options to a Test Suite pp.

Setting a Request Timeout

Analyzer 7.2.0 Regression test actions that do a network request have a timeout. You can change the timeout defaults via Java system properties:

Action

Default Timeout

Java System Property

Value

Establish connection

1 minute

sun.net.client.defaultConnectTimeout

Specify a value in milliseconds. A value of -1 means: no timeout.

Wait for response

10 minutes

sun.net.client.defaultReadTimeout

Specify a value in milliseconds. A value of -1 means: no timeout.

The following test actions do support timeouts:

Set the changed timeouts via the Regression Test Command Line Tool call like

CODE
java -Dsun.net.client.defaultReadTimeout=5000 -jar RegTestRunner.jar ...

The command above starts tests with a timeout of 5000 ms for waiting for a response of network requests.

To set the same timeouts for the Analyzer refer to Setting a Request Timeout.

Examples

To try out the Regression Test Command Line Tool, you can use the PState Builder project that is delivered with the Builder examples.

Click here to download the example file: Builder project Advanced Modeling/PState

<your example path>\Advanced Modeling\PState\uml\pstatePurchaseOrder.xml

Analyzer

Regression Test Command Line Tool

CODE
C:\E2E Documentation\Advanced Modeling>java -jar RegTestRunner.jar -project PState -list
/Build Tests
/QA Tests
/QA Tests/PurchaseOrderExample
/QA Tests/PurchaseOrderExample/createPurchaseOrder
/QA Tests/PurchaseOrderExample/addNewItem
/QA Tests/PurchaseOrderExample/addGratifications
/QA Tests/PurchaseOrderExample/getAllPurchaseOrders
/QA Tests/PurchaseOrderExample/getPurchaseOrder
/QA Tests/PurchaseOrderExample/checkOut
/QA Tests/PurchaseOrderExample/getPurchaseOrdersByDate
/QA Tests/PurchaseOrderExample/closePurchaseOrder
/Dev Tests
/Dev Tests/PurchaseOrderExample
/Dev Tests/PurchaseOrderExample/createPurchaseOrder
/Dev Tests/PurchaseOrderExample/addNewItem
/Dev Tests/PurchaseOrderExample/addGratifications
/Dev Tests/PurchaseOrderExample/getAllPurchaseOrders
/Dev Tests/PurchaseOrderExample/getPurchaseOrder
/Dev Tests/PurchaseOrderExample/checkOut
/Dev Tests/PurchaseOrderExample/getPurchaseOrdersByDate
/Dev Tests/PurchaseOrderExample/closePurchaseOrder

Analyzer

Regression Test Command Line Tool

CODE
C:\E2E Documentation\Advanced Modeling>java -jar RegTestRunner.jar -project PState -suite "Build Tests"
Running Test '<buildRepositoryTest>Build repository of PurchaseOrderExample'.
Finished: 19.094 seconds

<testsuites>
   <testsuite name="TestSuites.Build Tests" tests="1">
      <testcase errors="0" name="Build repository of PurchaseOrderExample" time="19.091"/>
   </testsuite>
</testsuites>

Analyzer

Regression Test Command Line Tool

CODE
C:\E2E Documentation\Advanced Modeling>java -jar RegTestRunner.jar -project PState -suite "QA Tests/PurchaseOrderExample"
Running Test '<startservice>Start PurchaseOrderExample'.
Running Test '<TestcaseTest>Create Purchase Order 1'.
Running Test '<TestcaseTest>Create Purchase Order 2'.
Running Test '<TestcaseTest>Add Item 1'.
Running Test '<TestcaseTest>Add Item 2'.
Running Test '<TestcaseTest>Add Gratifications for Wishes Unlimited'.
Running Test '<TestcaseTest>getAllPurchaseOrders'.
Running Test '<TestcaseTest>Get Purchase Order 1'.
Running Test '<TestcaseTest>Get Purchase Order 4'.
Running Test '<TestcaseTest>Check Out Order 1'.
Running Test '<TestcaseTest>Check Out Order 2'.
Running Test '<TestcaseTest>Get Purchase Orders from 1970-11-01'.
Running Test '<TestcaseTest>Close Purchase Order 1'.
Running Test '<TestcaseTest>Close Purchase Order 2'.
Running Test '<stopservice>Stop PurchaseOrderExample'.
Finished: 8.261 seconds
<testsuites>
   <testsuite name="TestSuites.QA Tests.PurchaseOrderExample" tests="15">
      <testcase errors="0" name="Start PurchaseOrderExample" time="1.198"/>
      <testcase errors="0" name="Stop PurchaseOrderExample" time="5.694"/>
   </testsuite>
   <testsuite name="TestSuites.QA Tests.PurchaseOrderExample.createPurchaseOrder" tests="2">
      <testcase errors="0" name="Create Purchase Order 1" time="0.316"/>
      <testcase errors="0" name="Create Purchase Order 2" time="0.116"/>
   </testsuite>
   <testsuite name="TestSuites.QA Tests.PurchaseOrderExample.addNewItem" tests="2">
      <testcase errors="0" name="Add Item 1" time="0.119"/>
      <testcase errors="1" name="Add Item 2" time="0.111">
         <error message="Assertion Failed Error - [Reference = D:\E2E Documentation 18\Advanced Modeling\PState\regressiontest\reference\libraries\PurchaseOrderExample\addNewItem\PurchaseOrderService.PurchaseOrderPort.addNewItem\Add Item 2\request.log]&#xA;&#xA;[different] Expected text value '1' but was '99' - comparing &lt;purchaseOrderID ...&gt;1&lt;/purchaseOrderID&gt; at /env:Envelope[1]/env:Body[1]/ns0:addNewItem[1]/purchaseOrderID[1]/text()[1] to &lt;purchaseOrderID ...&gt;99&lt;/purchaseOrderID&gt; at /env:Envelope[1]/env:Body[1]/ns0:addNewItem[1]/purchaseOrderID[1]/text()[1]&#xA;&#xA;[Reference = D:\E2E Documentation 18\Advanced Modeling\PState\regressiontest\reference\libraries\PurchaseOrderExample\addNewItem\PurchaseOrderService.PurchaseOrderPort.addNewItem\Add Item 2\response.log]&#xA;&#xA;[different] Expected attribute name 'xmlns:ns0' but was 'null' - comparing 
   [...]
</testsuites>

Format of the Test Logfile

The logfile generated by the -logfile parameter is formatted in a junit compatible xml format that is used by Jenkins and others:

XML
<testsuites>
   <testsuite name="TestSuites" tests="10"/>
   <testsuite name="TestSuites.Dev Tests" tests="10"/>
      <testsuite name="TestSuites.Dev Tests.PurchaseOrderExample" tests="10">
         <testcase errors="0" name="Create Purchase Order 1" time="0.854"/>
         <testcase errors="1" name="Create Purchase Order 2" time="0.01">
            <error message="Assertion Failed Error - [...]"/>
         </testcase>
         [...]
   </testsuite>
</testsuites>

The XML structure of the logfile reflects the test case structure in the Regression Test Tool and gives some additional information on the test execution:

XML Element

XML Attribute

Description

testsuites

-

Root element.

testsuite

name

Name of the test suite including test suite path in a test suite hierarchy.

tests

Number of tests in the test suite.

testcase

name

Name of the test case.

errors

Number of errors during test execution.

time

Elapsed test time in seconds.

error

message

Text of the test error message.

Troubleshooting: Bridge Certificate Exception

Problem

Running tests with the Regression Test Command Line Tool, Bridge calls fail and you are getting this error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Certificates does not conform to algorithm constraints.

Solution

Check the certificate on your Bridge (see Managing the Bridge Certificate). MD5 certificates are deprecated as insecure. If your Bridge still uses MD algorithm (MD5withRSA), you should create a new certificate and restart the Bridge. The new certificate should have algorithm SHA256withRSA.

Workaround:
If you can not fix the Bridge as described above (for whatever reason), you can change file <jre>/lib/security/java.security of the JRE you use to run the Regression Test Command Line Tool. Remove the "MD5" parts in jdk.certpath.disabledAlgorithms and jdk.tls.disabledAlgorithms to enable the usage of MD5 certificates.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.