Test Statistics

Total Statistics
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
All Tests
835000:01:14
Statistics by Tag
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
1node
835000:01:14
critical
835000:01:14
netconf
835000:01:14
Statistics by Suite
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
netconf-clustering-scale.txt
835000:01:26

Test Execution Log

00:01:25.821 SUITE netconf-clustering-scale.txt
Full Name: netconf-clustering-scale.txt
Documentation:

netconf-restperfclient update performance test suite (clustered setup).

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Perform given count of update operations on device data mounted onto a netconf connector (using the netconf-testtool-restperfclient tool) and see how much time it took. More exactly, it sends the data to a restconf mountpoint of the netconf connector belonging to the device, which turns out to turn the first request sent to a "create" request and the remaining requests to "update" requests (due to how the testtool device behavior is implemented).

The difference from the "single node" test suite (see ../scale/performance.robot) is that the device is configured and the data on it created using one node in the cluster and the update operations are issued on a different node. This forces the cluster nodes to communicate with each other about the data to be sent to the device.
Source: /w/workspace/netconf-csit-3node-clustering-scale-only-calcium/test/csit/suites/netconf/clusteringscale/performance.robot
Start / End / Elapsed: 20241117 22:56:10.327 / 20241117 22:57:36.148 / 00:01:25.821
Status: 8 tests total, 3 passed, 5 failed, 0 skipped
00:00:10.823 SETUP Setup_Everything  
Documentation:

Setup everything needed for the test cases.
Start / End / Elapsed: 20241117 22:56:10.892 / 20241117 22:56:21.715 / 00:00:10.823
00:00:00.345 TEARDOWN Teardown_Everything  
Documentation:

Teardown the test infrastructure, perform cleanup and release all resources.
Start / End / Elapsed: 20241117 22:57:35.803 / 20241117 22:57:36.148 / 00:00:00.345
00:00:08.370 TEST Start_Testtool
Full Name: netconf-clustering-scale.txt.Start_Testtool
Documentation:

Deploy and start test tool, then wait for all its devices to become online.
Tags: 1node, critical, netconf
Start / End / Elapsed: 20241117 22:56:21.715 / 20241117 22:56:30.085 / 00:00:08.370
Status: PASS
00:00:02.167 TEST Configure_Device_On_Netconf
Full Name: netconf-clustering-scale.txt.Configure_Device_On_Netconf
Documentation:

Configure the testtool device on Netconf connector, using node 1.
Tags: 1node, critical, netconf
Start / End / Elapsed: 20241117 22:56:30.086 / 20241117 22:56:32.253 / 00:00:02.167
Status: PASS
00:00:02.578 TEST Wait_For_Device_To_Become_Connected
Full Name: netconf-clustering-scale.txt.Wait_For_Device_To_Become_Connected
Documentation:

Wait until the device becomes available through Netconf on node 1.
Tags: 1node, critical, netconf
Start / End / Elapsed: 20241117 22:56:32.253 / 20241117 22:56:34.831 / 00:00:02.578
Status: PASS
00:01:00.658 TEST Wait_For_Device_Data_To_Be_Seen
Full Name: netconf-clustering-scale.txt.Wait_For_Device_Data_To_Be_Seen
Documentation:

Wait until the device data show up at node 2.
Tags: 1node, critical, netconf
Start / End / Elapsed: 20241117 22:56:34.831 / 20241117 22:57:35.489 / 00:01:00.658
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "netconf_clustering_scale_txt_wait_for_device_data_to_be_seen" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=netconf_clustering_scale_txt_wait_for_device_data_to_be_seen&order=bug_status" Keyword 'Check_Data_Present' failed after retrying for 1 minute. The last error was: HTTPError: 503 Server Error: Service Unavailable for url: http://10.30.170.165:8181/rests/data/network-topology:network-topology/topology=topology-netconf/node=17830-sim-device/yang-ext:mount?content=config
00:00:00.293 SETUP SetupUtils . Setup_Test_With_Logging_And_Fast_Failing  
Documentation:

Test case setup which skips on previous failure. If not, logs test case name to Karaf log. Recommended to be used as the default test case setup.
Start / End / Elapsed: 20241117 22:56:34.832 / 20241117 22:56:35.125 / 00:00:00.293
00:01:00.352 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${DEVICE_DATA_CONNECT_TIMEOUT} 1s Check_Data_Present
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20241117 22:56:35.125 / 20241117 22:57:35.477 / 00:01:00.352
Message: 57 failing items removed using the --remove-keywords option.
00:00:00.030 KEYWORD Check_Data_Present  
Start / End / Elapsed: 20241117 22:57:35.447 / 20241117 22:57:35.477 / 00:00:00.030
00:00:00.001 KEYWORD ${url} = BuiltIn . Set Variable   ${REST_API}/network-topology:network-topology/topology=topology-netconf/node=${DEVICE_NAME}/yang-ext:mount?content=config
Documentation:

Returns the given values which can then be assigned to a variables.
Start / End / Elapsed: 20241117 22:57:35.448 / 20241117 22:57:35.449 / 00:00:00.001
00:00:00.027 KEYWORD ${data} = TemplatedRequests . Get_As_Xml_From_Uri   ${url} session=node2
Documentation:

Specify XML headers and return Get_From_Uri response text.
Start / End / Elapsed: 20241117 22:57:35.449 / 20241117 22:57:35.476 / 00:00:00.027
00:00:00.021 KEYWORD ${response_text} = TemplatedRequests . Get_From_Uri   uri=${uri} accept=${ACCEPT_XML} session=${session} normalize_json=False http_timeout=${http_timeout} log_response=${log_response}
Documentation:

GET data from given URI, check status code and return response text. ${accept} is a Python object with headers to use. If ${normalize_json}, normalize as JSON text before returning.
Start / End / Elapsed: 20241117 22:57:35.455 / 20241117 22:57:35.476 / 00:00:00.021
00:00:00.000 KEYWORD BuiltIn . Log   ${uri}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20241117 22:57:35.464 / 20241117 22:57:35.464 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${accept}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20241117 22:57:35.464 / 20241117 22:57:35.464 / 00:00:00.000
00:00:00.010 IF """${http_timeout}""" == """${EMPTY}"""  
Start / End / Elapsed: 20241117 22:57:35.464 / 20241117 22:57:35.474 / 00:00:00.010
00:00:00.008 KEYWORD ${response} = RequestsLibrary . GET On Session   ${session} url=${uri} headers=${accept}
Documentation:

Sends a GET request on a previously created HTTP Session.
Start / End / Elapsed: 20241117 22:57:35.466 / 20241117 22:57:35.474 / 00:00:00.008
22:57:35.474 INFO GET Request : url=http://10.30.170.165:8181/rests/data/network-topology:network-topology/topology=topology-netconf/node=17830-sim-device/yang-ext:mount?content=config path_url=/rests/data/network-topology:network-topology/topology=topology-netconf/node=17830-sim-device/yang-ext:mount?content=config headers={'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/xml', 'Connection': 'keep-alive', 'Content-Type': 'application/xml', 'Cookie': 'JSESSIONID=node0up1p3citu38pjn6bpeki3g7f0.node0', 'Authorization': 'Basic YWRtaW46YWRtaW4='} body=None
22:57:35.474 INFO GET Response : url=http://10.30.170.165:8181/rests/data/network-topology:network-topology/topology=topology-netconf/node=17830-sim-device/yang-ext:mount?content=config status=503, reason=Service Unavailable headers={'Content-Type': 'application/yang-data+xml', 'Content-Length': '350'} body=<?xml version="1.0" ?> <errors xmlns="urn:ietf:params:xml:ns:yang:ietf-restconf"> <error> <error-type>protocol</error-type> <error-message>Mount point 'network-topology:network-topology/topology=topology-netconf/node=17830-sim-device' does not exist</error-message> <error-tag>resource-denied-transport</error-tag> </error> </errors>
22:57:35.474 FAIL HTTPError: 503 Server Error: Service Unavailable for url: http://10.30.170.165:8181/rests/data/network-topology:network-topology/topology=topology-netconf/node=17830-sim-device/yang-ext:mount?content=config
00:00:00.000 ELSE  
Start / End / Elapsed: 20241117 22:57:35.474 / 20241117 22:57:35.474 / 00:00:00.000
00:00:00.000 KEYWORD TemplatedRequests . Check_Status_Code   ${response} log_response=${log_response}
Documentation:

Log response text, check status_code is one of allowed ones. In cases where this keyword is called in a WUKS it could end up logging tons of data and it may be desired to skip the logging by passing log_response=False, but by default it remains True.
Start / End / Elapsed: 20241117 22:57:35.475 / 20241117 22:57:35.475 / 00:00:00.000
00:00:00.000 IF not ${normalize_json}  
Start / End / Elapsed: 20241117 22:57:35.475 / 20241117 22:57:35.475 / 00:00:00.000
00:00:00.000 KEYWORD ${text_normalized} = norm_json . Normalize Json Text   ${response.text} jmes_path=${jmes_path} keys_with_volatiles=${keys_with_volatiles}
Documentation:

Attempt to return sorted indented JSON string.
Start / End / Elapsed: 20241117 22:57:35.476 / 20241117 22:57:35.476 / 00:00:00.000
00:00:00.000 RETURN   ${text_normalized}
Start / End / Elapsed: 20241117 22:57:35.476 / 20241117 22:57:35.476 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20241117 22:57:35.476 / 20241117 22:57:35.476 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Should Be Equal As Strings   ${data} <data xmlns="${ODL_NETCONF_NAMESPACE}"></data>
Documentation:

Fails if objects are unequal after converting them to strings.
Start / End / Elapsed: 20241117 22:57:35.476 / 20241117 22:57:35.476 / 00:00:00.000
22:57:35.477 FAIL Keyword 'Check_Data_Present' failed after retrying for 1 minute. The last error was: HTTPError: 503 Server Error: Service Unavailable for url: http://10.30.170.165:8181/rests/data/network-topology:network-topology/topology=topology-netconf/node=17830-sim-device/yang-ext:mount?content=config
00:00:00.012 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_And_Start_Fast_Failing_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs and start fast failing in case of failure.
Start / End / Elapsed: 20241117 22:57:35.477 / 20241117 22:57:35.489 / 00:00:00.012
00:00:00.013 TEST Create_Device_Data
Full Name: netconf-clustering-scale.txt.Create_Device_Data
Documentation:

Send some sample test data into the device through node 2 and check that the request went OK.
Tags: 1node, critical, netconf
Start / End / Elapsed: 20241117 22:57:35.489 / 20241117 22:57:35.502 / 00:00:00.013
Status: FAIL
Message: Setup failed: SKIPPED due to a failure in a previous fundamental test case.
00:00:00.003 SETUP SetupUtils . Setup_Test_With_Logging_And_Fast_Failing  
Documentation:

Test case setup which skips on previous failure. If not, logs test case name to Karaf log. Recommended to be used as the default test case setup.
Start / End / Elapsed: 20241117 22:57:35.490 / 20241117 22:57:35.493 / 00:00:00.003
00:00:00.001 KEYWORD FailFast . Fail_This_Fast_On_Previous_Error  
Documentation:

Mark (immediately) this test case as failed when fast-fail is enabled in suite.
Start / End / Elapsed: 20241117 22:57:35.492 / 20241117 22:57:35.493 / 00:00:00.001
00:00:00.000 IF '''${SuiteFastFail}'''=='True'  
Start / End / Elapsed: 20241117 22:57:35.492 / 20241117 22:57:35.492 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Fail   SKIPPED due to a failure in a previous fundamental test case.
Documentation:

Fails the test with the given message and optionally alters its tags.
Start / End / Elapsed: 20241117 22:57:35.493 / 20241117 22:57:35.493 / 00:00:00.000
22:57:35.493 FAIL SKIPPED due to a failure in a previous fundamental test case.
00:00:00.000 KEYWORD BuiltIn . Run Keyword And Ignore Error   KarafKeywords.Log_Testcase_Start_To_Controller_Karaf member_index_list=${member_index_list}
Documentation:

Runs the given keyword with the given arguments and ignores possible error.
Start / End / Elapsed: 20241117 22:57:35.493 / 20241117 22:57:35.493 / 00:00:00.000
00:00:00.009 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_And_Start_Fast_Failing_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs and start fast failing in case of failure.
Start / End / Elapsed: 20241117 22:57:35.493 / 20241117 22:57:35.502 / 00:00:00.009
00:00:00.013 TEST Run_Restperfclient
Full Name: netconf-clustering-scale.txt.Run_Restperfclient
Documentation:

Deploy and execute restperfclient, asking it to send the specified amount of requests to the netconf connector of the device through node 3. The duration of this test case is the main performance metric.
Tags: 1node, critical, netconf
Start / End / Elapsed: 20241117 22:57:35.502 / 20241117 22:57:35.515 / 00:00:00.013
Status: FAIL
Message: Setup failed: SKIPPED due to a failure in a previous fundamental test case.
00:00:00.003 SETUP SetupUtils . Setup_Test_With_Logging_And_Fast_Failing  
Documentation:

Test case setup which skips on previous failure. If not, logs test case name to Karaf log. Recommended to be used as the default test case setup.
Start / End / Elapsed: 20241117 22:57:35.503 / 20241117 22:57:35.506 / 00:00:00.003
00:00:00.001 KEYWORD FailFast . Fail_This_Fast_On_Previous_Error  
Documentation:

Mark (immediately) this test case as failed when fast-fail is enabled in suite.
Start / End / Elapsed: 20241117 22:57:35.505 / 20241117 22:57:35.506 / 00:00:00.001
00:00:00.000 IF '''${SuiteFastFail}'''=='True'  
Start / End / Elapsed: 20241117 22:57:35.505 / 20241117 22:57:35.505 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Fail   SKIPPED due to a failure in a previous fundamental test case.
Documentation:

Fails the test with the given message and optionally alters its tags.
Start / End / Elapsed: 20241117 22:57:35.506 / 20241117 22:57:35.506 / 00:00:00.000
22:57:35.506 FAIL SKIPPED due to a failure in a previous fundamental test case.
00:00:00.000 KEYWORD BuiltIn . Run Keyword And Ignore Error   KarafKeywords.Log_Testcase_Start_To_Controller_Karaf member_index_list=${member_index_list}
Documentation:

Runs the given keyword with the given arguments and ignores possible error.
Start / End / Elapsed: 20241117 22:57:35.506 / 20241117 22:57:35.506 / 00:00:00.000
00:00:00.009 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_And_Start_Fast_Failing_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs and start fast failing in case of failure.
Start / End / Elapsed: 20241117 22:57:35.507 / 20241117 22:57:35.516 / 00:00:00.009
00:00:00.013 TEST Check_For_Failed_Requests
Full Name: netconf-clustering-scale.txt.Check_For_Failed_Requests
Documentation:

Make sure there are no failed requests in the restperfclient log. This is a separate test case to distinguish between restperfclient failure and failed requests. Failed requests are rejected because we don't want to test performance of ODL rejecting our requests. If this test case fails, then the duration of Run_Restperfclient cannot be trusted to show the real performance of the cluster.
Tags: 1node, critical, netconf
Start / End / Elapsed: 20241117 22:57:35.516 / 20241117 22:57:35.529 / 00:00:00.013
Status: FAIL
Message: Setup failed: SKIPPED due to a failure in a previous fundamental test case.
00:00:00.003 SETUP SetupUtils . Setup_Test_With_Logging_And_Fast_Failing  
Documentation:

Test case setup which skips on previous failure. If not, logs test case name to Karaf log. Recommended to be used as the default test case setup.
Start / End / Elapsed: 20241117 22:57:35.516 / 20241117 22:57:35.519 / 00:00:00.003
00:00:00.001 KEYWORD FailFast . Fail_This_Fast_On_Previous_Error  
Documentation:

Mark (immediately) this test case as failed when fast-fail is enabled in suite.
Start / End / Elapsed: 20241117 22:57:35.519 / 20241117 22:57:35.520 / 00:00:00.001
00:00:00.000 IF '''${SuiteFastFail}'''=='True'  
Start / End / Elapsed: 20241117 22:57:35.519 / 20241117 22:57:35.519 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Fail   SKIPPED due to a failure in a previous fundamental test case.
Documentation:

Fails the test with the given message and optionally alters its tags.
Start / End / Elapsed: 20241117 22:57:35.519 / 20241117 22:57:35.519 / 00:00:00.000
22:57:35.519 FAIL SKIPPED due to a failure in a previous fundamental test case.
00:00:00.000 KEYWORD BuiltIn . Run Keyword And Ignore Error   KarafKeywords.Log_Testcase_Start_To_Controller_Karaf member_index_list=${member_index_list}
Documentation:

Runs the given keyword with the given arguments and ignores possible error.
Start / End / Elapsed: 20241117 22:57:35.520 / 20241117 22:57:35.520 / 00:00:00.000
00:00:00.009 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_And_Start_Fast_Failing_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs and start fast failing in case of failure.
Start / End / Elapsed: 20241117 22:57:35.520 / 20241117 22:57:35.529 / 00:00:00.009
00:00:00.273 TEST Cleanup_And_Collect
Full Name: netconf-clustering-scale.txt.Cleanup_And_Collect
Documentation:

Deconfigure the testtool device on Netconf connector using node 1.
Tags: 1node, critical, netconf
Start / End / Elapsed: 20241117 22:57:35.529 / 20241117 22:57:35.802 / 00:00:00.273
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "netconf_clustering_scale_txt_cleanup_and_collect" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=netconf_clustering_scale_txt_cleanup_and_collect&order=bug_status" '' == ''
00:00:00.261 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20241117 22:57:35.529 / 20241117 22:57:35.790 / 00:00:00.261
00:00:00.001 KEYWORD RestPerfClient . Collect_From_Restperfclient  
Documentation:

Collect useful data produced by restperfclient
Start / End / Elapsed: 20241117 22:57:35.790 / 20241117 22:57:35.791 / 00:00:00.001
00:00:00.000 KEYWORD BuiltIn . Should Not Be Equal   '${RestPerfClient__restperfclientlog}' ''
Documentation:

Fails if the given objects are equal.
Start / End / Elapsed: 20241117 22:57:35.791 / 20241117 22:57:35.791 / 00:00:00.000
22:57:35.791 FAIL '' == ''
00:00:00.000 KEYWORD SSHLibrary . Get File   ${RestPerfClient__restperfclientlog}
Documentation:

Downloads file(s) from the remote machine to the local machine.
Start / End / Elapsed: 20241117 22:57:35.791 / 20241117 22:57:35.791 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Set Suite Variable   ${RestPerfClient__restperfclientlog} ${EMPTY}
Documentation:

Makes a variable available everywhere within the scope of the current suite.
Start / End / Elapsed: 20241117 22:57:35.791 / 20241117 22:57:35.791 / 00:00:00.000
00:00:00.000 KEYWORD NetconfKeywords . Remove_Device_From_Netconf   ${DEVICE_NAME} session=node1
Documentation:

Tell Netconf to deconfigure the specified device
Start / End / Elapsed: 20241117 22:57:35.792 / 20241117 22:57:35.792 / 00:00:00.000
00:00:00.010 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_And_Start_Fast_Failing_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs and start fast failing in case of failure.
Start / End / Elapsed: 20241117 22:57:35.792 / 20241117 22:57:35.802 / 00:00:00.010