Test Statistics

Total Statistics
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
All Tests
1037231000:29:23
Statistics by Tag
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
No Tags
Statistics by Suite
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
openflowplugin-clustering-bulkomatic.txt
1037231000:29:38
openflowplugin-clustering-bulkomatic . txt . Cluster Reconcilliation Multi DPN
18180000:01:06
openflowplugin-clustering-bulkomatic . txt . Cluster HA Data Recovery BulkFlow 2Node Cluster
30300000:03:05
openflowplugin-clustering-bulkomatic . txt . Cluster HA Data Recovery BulkFlow Single Switch
432419000:25:18
openflowplugin-clustering-bulkomatic . txt . Cluster Current Term Verification 3Node Cluster
12012000:00:08

Test Execution Log

00:29:38.028 SUITE openflowplugin-clustering-bulkomatic.txt
Full Name: openflowplugin-clustering-bulkomatic.txt
Source: /w/workspace/openflowplugin-csit-3node-clustering-bulkomatic-only-titanium/test/csit/suites/openflowplugin/Clustering_Bulkomatic
Start / End / Elapsed: 20250318 02:08:34.243 / 20250318 02:38:12.271 / 00:29:38.028
Status: 103 tests total, 72 passed, 31 failed, 0 skipped
00:01:06.474 SUITE Cluster Reconcilliation Multi DPN
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN
Documentation:

Test suite for Cluster with Bulk Flows - Reconcilliation in a multi DPN environment
Source: /w/workspace/openflowplugin-csit-3node-clustering-bulkomatic-only-titanium/test/csit/suites/openflowplugin/Clustering_Bulkomatic/010__Cluster_Reconcilliation_Multi_DPN.robot
Start / End / Elapsed: 20250318 02:08:34.266 / 20250318 02:09:40.740 / 00:01:06.474
Status: 18 tests total, 18 passed, 0 failed, 0 skipped
00:03:05.385 SUITE Cluster HA Data Recovery BulkFlow 2Node Cluster
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster
Documentation:

Test suite for 2Node Cluster HA with Bulk Flows - Cluster node convergance and Data consistency after leader and follower restart with one switch connected
Source: /w/workspace/openflowplugin-csit-3node-clustering-bulkomatic-only-titanium/test/csit/suites/openflowplugin/Clustering_Bulkomatic/020__Cluster_HA_Data_Recovery_BulkFlow_2Node_Cluster.robot
Start / End / Elapsed: 20250318 02:09:40.741 / 20250318 02:12:46.126 / 00:03:05.385
Status: 30 tests total, 30 passed, 0 failed, 0 skipped
00:25:18.087 SUITE Cluster HA Data Recovery BulkFlow Single Switch
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch
Documentation:

Test suite for Cluster HA with Bulk Flows - Data consistency after cluster restart, leader restart and follower restart with one switch connected
Source: /w/workspace/openflowplugin-csit-3node-clustering-bulkomatic-only-titanium/test/csit/suites/openflowplugin/Clustering_Bulkomatic/030__Cluster_HA_Data_Recovery_BulkFlow_Single_Switch.robot
Start / End / Elapsed: 20250318 02:12:46.127 / 20250318 02:38:04.214 / 00:25:18.087
Status: 43 tests total, 24 passed, 19 failed, 0 skipped
00:00:02.487 SETUP SetupUtils . Setup_Utils_For_Setup_And_Teardown  
Documentation:

Prepare both FailFast and Karaf logging, log that the caller suite has started. This keyword is to be used in suite setup, and without it the other keywords may not work properly.
Start / End / Elapsed: 20250318 02:12:46.197 / 20250318 02:12:48.684 / 00:00:02.487
00:00:00.000 TEARDOWN RequestsLibrary . Delete All Sessions  
Documentation:

Removes all the session objects
Start / End / Elapsed: 20250318 02:38:04.213 / 20250318 02:38:04.213 / 00:00:00.000
00:00:07.746 TEST Check Shards Status And Initialize Variables
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Check Shards Status And Initialize Variables
Documentation:

Check Status for all shards in OpenFlow application.
Start / End / Elapsed: 20250318 02:12:48.684 / 20250318 02:12:56.430 / 00:00:07.746
Status: PASS
00:00:01.118 TEST Get Inventory Follower Before Cluster Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Get Inventory Follower Before Cluster Restart
Documentation:

Find a follower in the inventory config shard
Start / End / Elapsed: 20250318 02:12:56.430 / 20250318 02:12:57.548 / 00:00:01.118
Status: PASS
00:00:03.307 TEST Start Mininet Connect To Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Start Mininet Connect To Follower Node1
Documentation:

Start mininet with connection to Follower Node1.
Start / End / Elapsed: 20250318 02:12:57.548 / 20250318 02:13:00.855 / 00:00:03.307
Status: PASS
00:00:01.518 TEST Add Bulk Flow From Follower
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Add Bulk Flow From Follower
Documentation:

1000 Flows added via Follower Node1 and verify it gets applied in all instances.
Start / End / Elapsed: 20250318 02:13:00.856 / 20250318 02:13:02.374 / 00:00:01.518
Status: PASS
00:00:03.043 TEST Get Bulk Flows and Verify In Cluster
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Get Bulk Flows and Verify In Cluster
Documentation:

Initiate get operation and check flow count across cluster nodes
Start / End / Elapsed: 20250318 02:13:02.374 / 20250318 02:13:05.417 / 00:00:03.043
Status: PASS
00:00:00.432 TEST Verify Flows In Switch Before Cluster Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify Flows In Switch Before Cluster Restart
Documentation:

Verify flows are installed in switch before cluster restart.
Start / End / Elapsed: 20250318 02:13:05.417 / 20250318 02:13:05.849 / 00:00:00.432
Status: PASS
00:00:06.498 TEST Kill All Cluster Nodes
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Kill All Cluster Nodes
Documentation:

Kill All Nodes.
Start / End / Elapsed: 20250318 02:13:05.849 / 20250318 02:13:12.347 / 00:00:06.498
Status: PASS
00:00:33.340 TEST Stop Mininet Connected To Follower Node1 and Exit
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Stop Mininet Connected To Follower Node1 and Exit
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250318 02:13:12.348 / 20250318 02:13:45.688 / 00:00:33.340
Status: PASS
00:00:40.249 TEST Restart All Cluster Nodes
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Restart All Cluster Nodes
Documentation:

Restart all cluster nodes.
Start / End / Elapsed: 20250318 02:13:45.687 / 20250318 02:14:25.936 / 00:00:40.249
Status: PASS
00:00:05.459 TEST Verify Data Recovery After Cluster Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify Data Recovery After Cluster Restart
Documentation:

1000 Flows preserved in all controller instances.
Start / End / Elapsed: 20250318 02:14:25.936 / 20250318 02:14:31.395 / 00:00:05.459
Status: PASS
00:00:03.628 TEST Start Mininet Again Connect To Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Start Mininet Again Connect To Follower Node1
Documentation:

Start mininet with connection to follower node1.
Start / End / Elapsed: 20250318 02:14:31.395 / 20250318 02:14:35.023 / 00:00:03.628
Status: PASS
00:00:02.616 TEST Verify Flows In Switch After Cluster Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify Flows In Switch After Cluster Restart
Documentation:

Verify flows are installed in switch after cluster restart.
Start / End / Elapsed: 20250318 02:14:35.023 / 20250318 02:14:37.639 / 00:00:02.616
Status: PASS
00:00:02.584 TEST Stop Mininet Connected To Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Stop Mininet Connected To Follower Node1
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250318 02:14:37.639 / 20250318 02:14:40.223 / 00:00:02.584
Status: PASS
00:00:01.546 TEST Delete All Flows From Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Delete All Flows From Follower Node1
Documentation:

1000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
Start / End / Elapsed: 20250318 02:14:40.223 / 20250318 02:14:41.769 / 00:00:01.546
Status: PASS
00:00:02.913 TEST Verify No Flows In Cluster
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify No Flows In Cluster
Documentation:

Verify flow count is 0 across cluster nodes.
Start / End / Elapsed: 20250318 02:14:41.769 / 20250318 02:14:44.682 / 00:00:02.913
Status: PASS
00:00:01.212 TEST Get Inventory Leader Before Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Get Inventory Leader Before Leader Restart
Documentation:

Find leader in the inventory config shard
Start / End / Elapsed: 20250318 02:14:44.682 / 20250318 02:14:45.894 / 00:00:01.212
Status: PASS
00:00:02.879 TEST Start Mininet Connect To Leader
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Start Mininet Connect To Leader
Documentation:

Start mininet with connection to Leader Node.
Start / End / Elapsed: 20250318 02:14:45.895 / 20250318 02:14:48.774 / 00:00:02.879
Status: PASS
00:00:01.561 TEST Add Bulk Flow From Leader
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Add Bulk Flow From Leader
Documentation:

1000 Flows added via Follower Node1 and verify it gets applied in all instances.
Start / End / Elapsed: 20250318 02:14:48.773 / 20250318 02:14:50.334 / 00:00:01.561
Status: PASS
00:00:03.010 TEST Get Bulk Flows and Verify In Cluster Before Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Get Bulk Flows and Verify In Cluster Before Leader Restart
Documentation:

Initiate get operation and check flow count across cluster nodes
Start / End / Elapsed: 20250318 02:14:50.335 / 20250318 02:14:53.345 / 00:00:03.010
Status: PASS
00:00:02.010 TEST Verify Flows In Switch Before Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify Flows In Switch Before Leader Restart
Documentation:

Verify flows are installed in switch before leader restart.
Start / End / Elapsed: 20250318 02:14:53.346 / 20250318 02:14:55.356 / 00:00:02.010
Status: PASS
00:00:14.053 TEST Kill Leader From Cluster Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Kill Leader From Cluster Node
Documentation:

Kill Leader Node.
Start / End / Elapsed: 20250318 02:14:55.356 / 20250318 02:15:09.409 / 00:00:14.053
Status: PASS
00:00:12.414 TEST Stop Mininet Connected To Leader Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Stop Mininet Connected To Leader Node
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250318 02:15:09.409 / 20250318 02:15:21.823 / 00:00:12.414
Status: PASS
00:05:31.942 TEST Restart Leader from Cluster Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Restart Leader from Cluster Node
Documentation:

Start Leader Node Up.
Start / End / Elapsed: 20250318 02:15:21.823 / 20250318 02:20:53.765 / 00:05:31.942
Status: FAIL
Message: Keyword 'Verify_Members_Are_Ready' failed after retrying for 5 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.40:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore
00:00:27.475 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250318 02:15:21.825 / 20250318 02:15:49.300 / 00:00:27.475
00:05:04.465 KEYWORD ClusterManagement . Start_Single_Member   ${Inventory_Leader}
Documentation:

Convenience keyword that starts the specified member of the cluster.
Start / End / Elapsed: 20250318 02:15:49.300 / 20250318 02:20:53.765 / 00:05:04.465
00:00:00.001 KEYWORD ${index_list} = ClusterManagement . ClusterManagement__Build_List   ${member}
Start / End / Elapsed: 20250318 02:15:49.303 / 20250318 02:15:49.304 / 00:00:00.001
00:00:00.002 KEYWORD ${member_ip} = ClusterManagement . Return_Member_IP   ${member}
Documentation:

Return the IP address of the member given the member_index.
Start / End / Elapsed: 20250318 02:15:49.305 / 20250318 02:15:49.307 / 00:00:00.002
00:00:00.002 KEYWORD ${msg} = BuiltIn . Set Variable If   "${msg}" == "${EMPTY}" Starting ODL${member} ${member_ip} Starting ODL${member} ${member_ip}, ${msg}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250318 02:15:49.307 / 20250318 02:15:49.309 / 00:00:00.002
00:00:00.133 KEYWORD KarafKeywords . Log_Message_To_Controller_Karaf   ${msg}
Documentation:

Make sure this resource is initialized. Send a message into the controller's karaf log file on every node listed (or all). By default, failure while processing a node is silently ignored, unless ${tolerate_failure} is False.
Start / End / Elapsed: 20250318 02:15:49.313 / 20250318 02:15:49.446 / 00:00:00.133
00:05:04.319 KEYWORD ClusterManagement . Start_Members_From_List_Or_All   ${index_list} ${wait_for_sync} ${timeout} check_system_status=${check_system_status} verify_restconf=${verify_restconf} service_list=${service_list}
Documentation:

If the list is empty, start all cluster members. Otherwise, start members based on present indices. If True, wait for cluster sync on listed members. Optionally karaf_home can be overriden. Optionally specific JAVA_HOME is used for starting. Garbage collection is unconditionally logged to files. TODO: Make that reasonable conditional?
Start / End / Elapsed: 20250318 02:15:49.447 / 20250318 02:20:53.766 / 00:05:04.319
00:00:00.000 KEYWORD ${base_command} = BuiltIn . Set Variable If   """${karaf_home}""" != "" ${karaf_home}/bin/start ${NODE_START_COMMAND}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250318 02:15:49.452 / 20250318 02:15:49.452 / 00:00:00.000
00:00:00.000 KEYWORD ${command} = BuiltIn . Set Variable If   """${export_java_home}""" != "" export JAVA_HOME="${export_java_home}"; ${base_command} ${base_command}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250318 02:15:49.452 / 20250318 02:15:49.452 / 00:00:00.000
00:00:00.000 KEYWORD ${epoch} = DateTime . Get Current Date   time_zone=UTC result_format=epoch exclude_millis=False
Documentation:

Returns current local or UTC time with an optional increment.
Start / End / Elapsed: 20250318 02:15:49.453 / 20250318 02:15:49.453 / 00:00:00.000
00:00:00.000 KEYWORD ${gc_filepath} = BuiltIn . Set Variable If   """${karaf_home}""" != "" ${karaf_home}/data/log/gc_${epoch}.log ${GC_LOG_PATH}/gc_${epoch}.log
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250318 02:15:49.453 / 20250318 02:15:49.453 / 00:00:00.000
00:00:00.000 KEYWORD ${gc_options} = BuiltIn . Set Variable If   "docker" not in """${node_start_command}""" -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:${gc_filepath} ${EMPTY}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250318 02:15:49.454 / 20250318 02:15:49.454 / 00:00:00.000
00:00:00.802 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=${command} ${gc_options} member_index_list=${member_index_list}
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250318 02:15:49.454 / 20250318 02:15:50.256 / 00:00:00.802
00:05:01.477 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${timeout} 10s Verify_Members_Are_Ready ${member_index_list} ${wait_for_sync} ${verify_restconf} ${check_system_status} ${service_list}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250318 02:15:50.256 / 20250318 02:20:51.733 / 00:05:01.477
Message: 30 failing items removed using the --remove-keywords option.
00:00:00.030 KEYWORD ClusterManagement . Verify_Members_Are_Ready   ${member_index_list} ${wait_for_sync} ${verify_restconf} ${check_system_status} ${service_list}
Documentation:

Verifies the specified readiness conditions for the given listed members after startup. If ${verify_cluster_sync}, verifies the datastores have synced with the rest of the cluster. If True, verifies RESTCONF is available. If ${verify_system_status}, verifies the system services are OPERATIONAL.
Start / End / Elapsed: 20250318 02:20:51.703 / 20250318 02:20:51.733 / 00:00:00.030
00:00:00.028 IF ${verify_cluster_sync}  
Start / End / Elapsed: 20250318 02:20:51.704 / 20250318 02:20:51.732 / 00:00:00.028
00:00:00.028 KEYWORD ClusterManagement . Check_Cluster_Is_In_Sync   ${member_index_list}
Documentation:

Fail if no-sync is detected on a member from list (or any).
Start / End / Elapsed: 20250318 02:20:51.704 / 20250318 02:20:51.732 / 00:00:00.028
00:00:00.001 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All   given_list=${member_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250318 02:20:51.705 / 20250318 02:20:51.706 / 00:00:00.001
00:00:00.026 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250318 02:20:51.706 / 20250318 02:20:51.732 / 00:00:00.026
00:00:00.025 ITERATION ${index} = 3  
Start / End / Elapsed: 20250318 02:20:51.706 / 20250318 02:20:51.731 / 00:00:00.025
00:00:00.024 KEYWORD ${status} = ClusterManagement . Get_Sync_Status_Of_Member   member_index=${index}
Documentation:

Obtain IP, two GETs from jolokia URIs, return combined sync status as string.
Start / End / Elapsed: 20250318 02:20:51.706 / 20250318 02:20:51.730 / 00:00:00.024
00:00:00.001 KEYWORD ${session} = ClusterManagement . Resolve_Http_Session_For_Member   member_index=${member_index}
Documentation:

Return RequestsLibrary session alias pointing to node of given index.
Start / End / Elapsed: 20250318 02:20:51.707 / 20250318 02:20:51.708 / 00:00:00.001
00:00:00.022 KEYWORD ${conf_text} = TemplatedRequests . Get_As_Json_From_Uri   uri=${JOLOKIA_CONF_SHARD_MANAGER_URI} session=${session}
Documentation:

Specify JSON headers and return Get_From_Uri normalized response text.
Start / End / Elapsed: 20250318 02:20:51.708 / 20250318 02:20:51.730 / 00:00:00.022
00:00:00.019 KEYWORD ${response_text} = TemplatedRequests . Get_From_Uri   uri=${uri} accept=${ACCEPT_EMPTY} session=${session} normalize_json=True http_timeout=${http_timeout} log_response=${log_response}
Documentation:

GET data from given URI, check status code and return response text. ${accept} is a Python object with headers to use. If ${normalize_json}, normalize as JSON text before returning.
Start / End / Elapsed: 20250318 02:20:51.711 / 20250318 02:20:51.730 / 00:00:00.019
00:00:00.000 KEYWORD BuiltIn . Log   ${uri}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250318 02:20:51.715 / 20250318 02:20:51.715 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${accept}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250318 02:20:51.715 / 20250318 02:20:51.715 / 00:00:00.000
00:00:00.013 IF """${http_timeout}""" == """${EMPTY}"""  
Start / End / Elapsed: 20250318 02:20:51.715 / 20250318 02:20:51.728 / 00:00:00.013
00:00:00.010 KEYWORD ${response} = RequestsLibrary . GET On Session   ${session} url=${uri} headers=${accept}
Documentation:

Sends a GET request on a previously created HTTP Session.
Start / End / Elapsed: 20250318 02:20:51.718 / 20250318 02:20:51.728 / 00:00:00.010
02:20:51.727 INFO GET Request : url=http://10.30.171.40:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore path_url=/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore headers={'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': 'Basic YWRtaW46YWRtaW4='} body=None
02:20:51.727 INFO GET Response : url=http://10.30.171.40:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore status=401, reason=Unauthorized headers={'Cache-Control': 'must-revalidate,no-cache,no-store', 'Content-Type': 'text/html;charset=iso-8859-1', 'Content-Length': '524'} body=<html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401 Unauthorized</h2> <table> <tr><th>URI:</th><td>/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>Unauthorized</td></tr> <tr><th>SERVLET:</th><td>org.jolokia.osgi.servlet.JolokiaServlet</td></tr> </table> </body> </html>
02:20:51.727 FAIL HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.40:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore
00:00:00.000 ELSE  
Start / End / Elapsed: 20250318 02:20:51.728 / 20250318 02:20:51.728 / 00:00:00.000
00:00:00.000 KEYWORD TemplatedRequests . Check_Status_Code   ${response} log_response=${log_response}
Documentation:

Log response text, check status_code is one of allowed ones. In cases where this keyword is called in a WUKS it could end up logging tons of data and it may be desired to skip the logging by passing log_response=False, but by default it remains True.
Start / End / Elapsed: 20250318 02:20:51.729 / 20250318 02:20:51.729 / 00:00:00.000
00:00:00.000 IF not ${normalize_json}  
Start / End / Elapsed: 20250318 02:20:51.729 / 20250318 02:20:51.729 / 00:00:00.000
00:00:00.000 KEYWORD ${text_normalized} = norm_json . Normalize Json Text   ${response.text} jmes_path=${jmes_path} keys_with_volatiles=${keys_with_volatiles}
Documentation:

Attempt to return sorted indented JSON string.
Start / End / Elapsed: 20250318 02:20:51.729 / 20250318 02:20:51.729 / 00:00:00.000
00:00:00.000 RETURN   ${text_normalized}
Start / End / Elapsed: 20250318 02:20:51.729 / 20250318 02:20:51.729 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250318 02:20:51.730 / 20250318 02:20:51.730 / 00:00:00.000
00:00:00.000 KEYWORD ${conf_status} = ClusterManagement . ClusterManagement__Parse_Sync_Status   shard_manager_text=${conf_text}
Documentation:

Return sync status parsed out of given text. Called twice by Get_Sync_Status_Of_Member.
Start / End / Elapsed: 20250318 02:20:51.730 / 20250318 02:20:51.730 / 00:00:00.000
00:00:00.000 IF 'False' == ${conf_status}  
Start / End / Elapsed: 20250318 02:20:51.730 / 20250318 02:20:51.730 / 00:00:00.000
00:00:00.000 KEYWORD ${oper_text} = TemplatedRequests . Get_As_Json_From_Uri   uri=${JOLOKIA_OPER_SHARD_MANAGER_URI} session=${session}
Documentation:

Specify JSON headers and return Get_From_Uri normalized response text.
Start / End / Elapsed: 20250318 02:20:51.730 / 20250318 02:20:51.730 / 00:00:00.000
00:00:00.000 KEYWORD ${oper_status} = ClusterManagement . ClusterManagement__Parse_Sync_Status   shard_manager_text=${oper_text}
Documentation:

Return sync status parsed out of given text. Called twice by Get_Sync_Status_Of_Member.
Start / End / Elapsed: 20250318 02:20:51.731 / 20250318 02:20:51.731 / 00:00:00.000
00:00:00.000 RETURN   ${oper_status}
Start / End / Elapsed: 20250318 02:20:51.731 / 20250318 02:20:51.731 / 00:00:00.000
00:00:00.000 IF 'True' == '${status}'  
Start / End / Elapsed: 20250318 02:20:51.731 / 20250318 02:20:51.731 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Fail   Index ${index} has incorrect status: ${status}
Documentation:

Fails the test with the given message and optionally alters its tags.
Start / End / Elapsed: 20250318 02:20:51.731 / 20250318 02:20:51.731 / 00:00:00.000
00:00:00.000 IF ${verify_restconf}  
Start / End / Elapsed: 20250318 02:20:51.732 / 20250318 02:20:51.732 / 00:00:00.000
00:00:00.000 IF ${verify_system_status} and ("${service_list}" != "[[]]")  
Start / End / Elapsed: 20250318 02:20:51.733 / 20250318 02:20:51.733 / 00:00:00.000
02:20:51.733 FAIL Keyword 'Verify_Members_Are_Ready' failed after retrying for 5 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.40:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore
00:00:02.031 TEARDOWN ClusterManagement . Run_Bash_Command_On_List_Or_All   command=netstat -pnatu | grep 2550
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250318 02:20:51.734 / 20250318 02:20:53.765 / 00:00:02.031
00:05:52.450 TEST Verify Data Recovery After Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify Data Recovery After Leader Restart
Documentation:

1000 Flows preserved in all controller instances.
Start / End / Elapsed: 20250318 02:20:53.766 / 20250318 02:26:46.216 / 00:05:52.450
Status: FAIL
Message: Keyword 'BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster' failed after retrying for 5 minutes 50 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.21:8181/rests/operations/sal-bulk-flow:read-flow-test
00:00:01.370 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250318 02:20:53.767 / 20250318 02:20:55.137 / 00:00:01.370
00:05:51.080 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${restart_timeout} 2s BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250318 02:20:55.136 / 20250318 02:26:46.216 / 00:05:51.080
Message: 157 failing items removed using the --remove-keywords option.
00:00:00.055 KEYWORD BulkomaticKeywords . Get Bulk Flow And Verify Count In Cluster   ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add}
Documentation:

Get Bulk Flow and Verify Flow Count in ${controller_index_list} matches ${flow_count}.
Start / End / Elapsed: 20250318 02:26:46.160 / 20250318 02:26:46.215 / 00:00:00.055
00:00:00.002 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All   given_list=${controller_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250318 02:26:46.163 / 20250318 02:26:46.165 / 00:00:00.002
00:00:00.038 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250318 02:26:46.165 / 20250318 02:26:46.203 / 00:00:00.038
00:00:00.038 ITERATION ${index} = 1  
Start / End / Elapsed: 20250318 02:26:46.165 / 20250318 02:26:46.203 / 00:00:00.038
00:00:00.034 KEYWORD BulkomaticKeywords . Get Bulk Flow   ${json_body_get} ${index}
Documentation:

Get Bulk Flow in member ${controller_index} according to ${json_body_get}.
Start / End / Elapsed: 20250318 02:26:46.170 / 20250318 02:26:46.204 / 00:00:00.034
00:00:00.030 KEYWORD ${resp} = ClusterManagement . Post_As_Json_To_Member   ${GET_BULK_CONFIG_NODES_API} ${json_body_get} ${controller_index}
Documentation:

Send a POST with the supplied uri and data to member ${member_index}.
Start / End / Elapsed: 20250318 02:26:46.174 / 20250318 02:26:46.204 / 00:00:00.030
00:00:00.001 KEYWORD ${session} = ClusterManagement . Resolve_Http_Session_For_Member   member_index=${member_index}
Documentation:

Return RequestsLibrary session alias pointing to node of given index.
Start / End / Elapsed: 20250318 02:26:46.174 / 20250318 02:26:46.175 / 00:00:00.001
00:00:00.021 KEYWORD ${response_text} = TemplatedRequests . Post_As_Json_To_Uri   uri=${uri} data=${data} session=${session}
Documentation:

Specify JSON headers and return Post_To_Uri normalized response text. Yang json content type is used as a workaround to RequestsLibrary json conversion eagerness. Response status code must be one of values from ${explicit_status_codes} if specified or one of set created from all positive HTTP status codes together with ${additional_allowed_status_codes}.
Start / End / Elapsed: 20250318 02:26:46.182 / 20250318 02:26:46.203 / 00:00:00.021
00:00:00.018 KEYWORD ${response_text} = TemplatedRequests . Post_To_Uri   uri=${uri} data=${data} accept=${ACCEPT_EMPTY} content_type=${HEADERS_YANG_JSON} session=${session} normalize_json=True additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes} http_timeout=${http_timeout}
Documentation:

POST data to given URI, check status code and return response text. ${content_type} and ${accept} are mandatory Python objects with headers to use. If ${normalize_json}, normalize text before returning.
Start / End / Elapsed: 20250318 02:26:46.184 / 20250318 02:26:46.202 / 00:00:00.018
00:00:00.000 KEYWORD BuiltIn . Log   ${uri}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250318 02:26:46.187 / 20250318 02:26:46.187 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${data}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250318 02:26:46.187 / 20250318 02:26:46.187 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${content_type}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250318 02:26:46.188 / 20250318 02:26:46.188 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${accept}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250318 02:26:46.188 / 20250318 02:26:46.188 / 00:00:00.000
00:00:00.002 KEYWORD ${headers} = TemplatedRequests . Join_Two_Headers   first=${content_type} second=${accept}
Documentation:

Take two dicts, join them, return result. Second argument values take precedence.
Start / End / Elapsed: 20250318 02:26:46.188 / 20250318 02:26:46.190 / 00:00:00.002
00:00:00.011 IF """${http_timeout}""" == """${EMPTY}"""  
Start / End / Elapsed: 20250318 02:26:46.190 / 20250318 02:26:46.201 / 00:00:00.011
00:00:00.009 KEYWORD ${response} = RequestsLibrary . POST On Session   ${session} ${uri} data=${data} headers=${headers}
Documentation:

Sends a POST request on a previously created HTTP Session.
Start / End / Elapsed: 20250318 02:26:46.192 / 20250318 02:26:46.201 / 00:00:00.009
02:26:46.200 INFO POST Request : url=http://10.30.170.21:8181/rests/operations/sal-bulk-flow:read-flow-test path_url=/rests/operations/sal-bulk-flow:read-flow-test headers={'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Cookie': 'JSESSIONID=node0c5gbzmvvmygxivyb6kulcx790.node0', 'Content-Length': '260', 'Authorization': 'Basic YWRtaW46YWRtaW4='} body={ "input" : { "sal-bulk-flow:verbose" : "false", "sal-bulk-flow:dpn-count" : "1", "sal-bulk-flow:is-config-ds" : "true", "sal-bulk-flow:flows-per-dpn" : "1000", "sal-bulk-flow:start-table-id" : "0", "sal-bulk-flow:end-table-id" : "1" } }
02:26:46.200 INFO POST Response : url=http://10.30.170.21:8181/rests/operations/sal-bulk-flow:read-flow-test status=401, reason=Unauthorized headers={'Cache-Control': 'must-revalidate,no-cache,no-store', 'Content-Type': 'text/html;charset=iso-8859-1', 'Content-Length': '417'} body=<html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401 Unauthorized</h2> <table> <tr><th>URI:</th><td>/rests/operations/sal-bulk-flow:read-flow-test</td></tr> <tr><th>STATUS:</th><td>401</td></tr> <tr><th>MESSAGE:</th><td>Unauthorized</td></tr> <tr><th>SERVLET:</th><td>default</td></tr> </table> </body> </html>
02:26:46.201 FAIL HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.21:8181/rests/operations/sal-bulk-flow:read-flow-test
00:00:00.000 ELSE  
Start / End / Elapsed: 20250318 02:26:46.201 / 20250318 02:26:46.201 / 00:00:00.000
00:00:00.000 KEYWORD TemplatedRequests . Check_Status_Code   ${response} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes}
Documentation:

Log response text, check status_code is one of allowed ones. In cases where this keyword is called in a WUKS it could end up logging tons of data and it may be desired to skip the logging by passing log_response=False, but by default it remains True.
Start / End / Elapsed: 20250318 02:26:46.202 / 20250318 02:26:46.202 / 00:00:00.000
00:00:00.000 IF not ${normalize_json}  
Start / End / Elapsed: 20250318 02:26:46.202 / 20250318 02:26:46.202 / 00:00:00.000
00:00:00.000 KEYWORD ${text_normalized} = norm_json . Normalize Json Text   ${response.text} jmes_path=${jmes_path}
Documentation:

Attempt to return sorted indented JSON string.
Start / End / Elapsed: 20250318 02:26:46.203 / 20250318 02:26:46.203 / 00:00:00.000
00:00:00.000 RETURN   ${text_normalized}
Start / End / Elapsed: 20250318 02:26:46.203 / 20250318 02:26:46.203 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250318 02:26:46.203 / 20250318 02:26:46.203 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250318 02:26:46.203 / 20250318 02:26:46.203 / 00:00:00.000
00:00:00.008 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250318 02:26:46.204 / 20250318 02:26:46.212 / 00:00:00.008
00:00:00.004 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250318 02:26:46.211 / 20250318 02:26:46.215 / 00:00:00.004
02:26:46.216 FAIL Keyword 'BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster' failed after retrying for 5 minutes 50 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.21:8181/rests/operations/sal-bulk-flow:read-flow-test
00:00:03.357 TEST Start Mininet Again Connect To Leader
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Start Mininet Again Connect To Leader
Documentation:

Start mininet with connection to Leader Node.
Start / End / Elapsed: 20250318 02:26:46.216 / 20250318 02:26:49.573 / 00:00:03.357
Status: PASS
00:01:42.465 TEST Verify Flows In Switch After Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify Flows In Switch After Leader Restart
Documentation:

Verify flows are installed in switch after leader restart.
Start / End / Elapsed: 20250318 02:26:49.574 / 20250318 02:28:32.039 / 00:01:42.465
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 1000.0
00:00:00.492 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250318 02:26:49.574 / 20250318 02:26:50.066 / 00:00:00.492
00:01:41.972 KEYWORD MininetKeywords . Verify Aggregate Flow From Mininet Session   ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
Documentation:

Verify flow count per switch
Start / End / Elapsed: 20250318 02:26:50.066 / 20250318 02:28:32.038 / 00:01:41.972
00:01:41.972 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${time_out} 2s MininetKeywords.Check Flows In Mininet ${mininet_conn} ${flow_count}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250318 02:26:50.067 / 20250318 02:28:32.039 / 00:01:41.972
Message: 50 failing items removed using the --remove-keywords option.
00:00:00.038 KEYWORD MininetKeywords . Check Flows In Mininet   ${mininet_conn} ${flow_count}
Documentation:

Sync with mininet to match exact number of flows
Start / End / Elapsed: 20250318 02:28:32.000 / 20250318 02:28:32.038 / 00:00:00.038
00:00:00.001 IF """${mininet_conn}""" != ""  
Start / End / Elapsed: 20250318 02:28:32.000 / 20250318 02:28:32.001 / 00:00:00.001
00:00:00.000 KEYWORD ${cmd} = BuiltIn . Set Variable   dpctl dump-aggregate -O OpenFlow13
Documentation:

Returns the given values which can then be assigned to a variables.
Start / End / Elapsed: 20250318 02:28:32.002 / 20250318 02:28:32.002 / 00:00:00.000
00:00:00.034 KEYWORD ${output} = MininetKeywords . Send Mininet Command   ${mininet_conn} ${cmd}
Documentation:

Sends Command dpctl dump-aggregate -O OpenFlow13 to Mininet session 387 and returns read buffer response.
Start / End / Elapsed: 20250318 02:28:32.003 / 20250318 02:28:32.037 / 00:00:00.034
00:00:00.000 KEYWORD ${flows} = String . Get Regexp Matches   ${output} (?<=flow_count\=).*?(?=\r)
Documentation:

Returns a list of all non-overlapping matches in the given string.
Start / End / Elapsed: 20250318 02:28:32.037 / 20250318 02:28:32.037 / 00:00:00.000
00:00:00.000 KEYWORD ${total_flows} = BuiltIn . Evaluate   sum(map(int, ${flows}))
Documentation:

Evaluates the given expression in Python and returns the result.
Start / End / Elapsed: 20250318 02:28:32.037 / 20250318 02:28:32.037 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Should Be Equal As Numbers   ${total_flows} ${flow_count}
Documentation:

Fails if objects are unequal after converting them to real numbers.
Start / End / Elapsed: 20250318 02:28:32.038 / 20250318 02:28:32.038 / 00:00:00.000
02:28:32.038 INFO Argument types are: <class 'int'> <class 'str'>
02:28:32.038 FAIL 0.0 != 1000.0
02:28:32.038 FAIL Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 1000.0
00:00:02.802 TEST Stop Mininet Connected To Leader Node After Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Stop Mininet Connected To Leader Node After Leader Restart
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250318 02:28:32.039 / 20250318 02:28:34.841 / 00:00:02.802
Status: PASS
00:00:00.465 TEST Delete All Flows From Leader Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Delete All Flows From Leader Node
Documentation:

1000 Flows deleted via Leader Node and verify it gets applied in all instances.
Start / End / Elapsed: 20250318 02:28:34.841 / 20250318 02:28:35.306 / 00:00:00.465
Status: FAIL
Message: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.40:8181/rests/operations/sal-bulk-flow:flow-test
00:00:00.415 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250318 02:28:34.842 / 20250318 02:28:35.257 / 00:00:00.415
00:00:00.044 KEYWORD BulkomaticKeywords . Delete Bulk Flow In Node   ${temp_json_config_del} ${Inventory_Leader} ${operation_timeout}
Documentation:

Delete Bulk Flow in member ${controller_index} and wait until operation is completed.
Start / End / Elapsed: 20250318 02:28:35.262 / 20250318 02:28:35.306 / 00:00:00.044
00:00:00.043 KEYWORD BulkomaticKeywords . Delete Bulk Flow   ${json_body_del} ${controller_index}
Documentation:

Delete Bulk Flow in member 3 according to ${json_body_del}.
Start / End / Elapsed: 20250318 02:28:35.262 / 20250318 02:28:35.305 / 00:00:00.043
00:00:00.039 KEYWORD ${resp} = ClusterManagement . Post_As_Json_To_Member   ${ADD_BULK_CONFIG_NODES_API} ${json_body_del} ${controller_index}
Documentation:

Send a POST with the supplied uri and data to member ${member_index}.
Start / End / Elapsed: 20250318 02:28:35.266 / 20250318 02:28:35.305 / 00:00:00.039
00:00:00.001 KEYWORD ${session} = ClusterManagement . Resolve_Http_Session_For_Member   member_index=${member_index}
Documentation:

Return RequestsLibrary session alias pointing to node of given index.
Start / End / Elapsed: 20250318 02:28:35.267 / 20250318 02:28:35.268 / 00:00:00.001
00:00:00.030 KEYWORD ${response_text} = TemplatedRequests . Post_As_Json_To_Uri   uri=${uri} data=${data} session=${session}
Documentation:

Specify JSON headers and return Post_To_Uri normalized response text. Yang json content type is used as a workaround to RequestsLibrary json conversion eagerness. Response status code must be one of values from ${explicit_status_codes} if specified or one of set created from all positive HTTP status codes together with ${additional_allowed_status_codes}.
Start / End / Elapsed: 20250318 02:28:35.275 / 20250318 02:28:35.305 / 00:00:00.030
00:00:00.027 KEYWORD ${response_text} = TemplatedRequests . Post_To_Uri   uri=${uri} data=${data} accept=${ACCEPT_EMPTY} content_type=${HEADERS_YANG_JSON} session=${session} normalize_json=True additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes} http_timeout=${http_timeout}
Documentation:

POST data to given URI, check status code and return response text. ${content_type} and ${accept} are mandatory Python objects with headers to use. If ${normalize_json}, normalize text before returning.
Start / End / Elapsed: 20250318 02:28:35.277 / 20250318 02:28:35.304 / 00:00:00.027
00:00:00.000 KEYWORD BuiltIn . Log   ${uri}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250318 02:28:35.279 / 20250318 02:28:35.279 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${data}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250318 02:28:35.280 / 20250318 02:28:35.280 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${content_type}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250318 02:28:35.280 / 20250318 02:28:35.280 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${accept}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250318 02:28:35.280 / 20250318 02:28:35.280 / 00:00:00.000
00:00:00.002 KEYWORD ${headers} = TemplatedRequests . Join_Two_Headers   first=${content_type} second=${accept}
Documentation:

Take two dicts, join them, return result. Second argument values take precedence.
Start / End / Elapsed: 20250318 02:28:35.281 / 20250318 02:28:35.283 / 00:00:00.002
00:00:00.019 IF """${http_timeout}""" == """${EMPTY}"""  
Start / End / Elapsed: 20250318 02:28:35.282 / 20250318 02:28:35.301 / 00:00:00.019
00:00:00.001 ELSE  
Start / End / Elapsed: 20250318 02:28:35.302 / 20250318 02:28:35.303 / 00:00:00.001
00:00:00.000 KEYWORD TemplatedRequests . Check_Status_Code   ${response} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes}
Documentation:

Log response text, check status_code is one of allowed ones. In cases where this keyword is called in a WUKS it could end up logging tons of data and it may be desired to skip the logging by passing log_response=False, but by default it remains True.
Start / End / Elapsed: 20250318 02:28:35.303 / 20250318 02:28:35.303 / 00:00:00.000
00:00:00.000 IF not ${normalize_json}  
Start / End / Elapsed: 20250318 02:28:35.303 / 20250318 02:28:35.303 / 00:00:00.000
00:00:00.000 KEYWORD ${text_normalized} = norm_json . Normalize Json Text   ${response.text} jmes_path=${jmes_path}
Documentation:

Attempt to return sorted indented JSON string.
Start / End / Elapsed: 20250318 02:28:35.304 / 20250318 02:28:35.304 / 00:00:00.000
00:00:00.000 RETURN   ${text_normalized}
Start / End / Elapsed: 20250318 02:28:35.304 / 20250318 02:28:35.304 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250318 02:28:35.304 / 20250318 02:28:35.304 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250318 02:28:35.305 / 20250318 02:28:35.305 / 00:00:00.000
00:00:00.000 KEYWORD BulkomaticKeywords . Wait Until Write Finishes   ${controller_index} ${timeout}
Documentation:

Wait Until Write operation status is OK in member 3.
Start / End / Elapsed: 20250318 02:28:35.305 / 20250318 02:28:35.305 / 00:00:00.000
00:00:00.484 TEST Verify No Flows In Cluster After Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify No Flows In Cluster After Leader Restart
Documentation:

Verify flow count is 0 across cluster nodes.
Start / End / Elapsed: 20250318 02:28:35.306 / 20250318 02:28:35.790 / 00:00:00.484
Status: FAIL
Message: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.21:8181/rests/operations/sal-bulk-flow:read-flow-test
00:00:11.088 TEST Get Inventory Follower Before follower Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Get Inventory Follower Before follower Restart
Documentation:

Find follower in the inventory config shard
Start / End / Elapsed: 20250318 02:28:35.791 / 20250318 02:28:46.879 / 00:00:11.088
Status: FAIL
Message: Keyword 'ClusterManagement.Get_Leader_And_Followers_For_Shard' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.21:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:00.425 TEST Start Mininet Connect To Follower Node2
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Start Mininet Connect To Follower Node2
Documentation:

Start mininet with connection to Follower Node2.
Start / End / Elapsed: 20250318 02:28:46.880 / 20250318 02:28:47.305 / 00:00:00.425
Status: FAIL
Message: Variable '${Follower_Node_2}' not found. Did you mean: ${Follower_Node_1}
00:00:00.412 TEST Add Bulk Flow From Follower Node2
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Add Bulk Flow From Follower Node2
Documentation:

1000 Flows added via Follower Node2 and verify it gets applied in all instances.
Start / End / Elapsed: 20250318 02:28:47.305 / 20250318 02:28:47.717 / 00:00:00.412
Status: FAIL
Message: Variable '${Follower_Node_2}' not found. Did you mean: ${Follower_Node_1}
00:00:00.522 TEST Get Bulk Flows and Verify In Cluster Before Follower Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Get Bulk Flows and Verify In Cluster Before Follower Restart
Documentation:

Initiate get operation and check flow count across cluster nodes.
Start / End / Elapsed: 20250318 02:28:47.717 / 20250318 02:28:48.239 / 00:00:00.522
Status: FAIL
Message: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.21:8181/rests/operations/sal-bulk-flow:read-flow-test
00:01:41.030 TEST Verify Flows In Switch Before Follower Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify Flows In Switch Before Follower Restart
Documentation:

Verify flows are installed in switch before follower restart.
Start / End / Elapsed: 20250318 02:28:48.240 / 20250318 02:30:29.270 / 00:01:41.030
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: Non-existing index or alias '387'.
00:00:00.376 TEST Kill Follower Node2
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Kill Follower Node2
Documentation:

Kill Follower Node2.
Start / End / Elapsed: 20250318 02:30:29.270 / 20250318 02:30:29.646 / 00:00:00.376
Status: FAIL
Message: Variable '${Follower_Node_2}' not found. Did you mean: ${Follower_Node_1}
00:00:00.484 TEST Stop Mininet Connected To Follower Node2 and Exit
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Stop Mininet Connected To Follower Node2 and Exit
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250318 02:30:29.647 / 20250318 02:30:30.131 / 00:00:00.484
Status: FAIL
Message: Non-existing index or alias '387'.
00:00:00.332 TEST Restart Follower Node2
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Restart Follower Node2
Documentation:

Start Follower Node2 Up.
Start / End / Elapsed: 20250318 02:30:30.131 / 20250318 02:30:30.463 / 00:00:00.332
Status: FAIL
Message: Variable '${Follower_Node_2}' not found. Did you mean: ${Follower_Node_1}
00:05:51.070 TEST Verify Data Recovery After Follower Node2 Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify Data Recovery After Follower Node2 Restart
Documentation:

1000 Flows preserved in all controller instances.
Start / End / Elapsed: 20250318 02:30:30.463 / 20250318 02:36:21.533 / 00:05:51.070
Status: FAIL
Message: Keyword 'BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster' failed after retrying for 5 minutes 50 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.21:8181/rests/operations/sal-bulk-flow:read-flow-test
00:00:00.419 TEST Start Mininet Again Connect To Follower Node2
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Start Mininet Again Connect To Follower Node2
Documentation:

Start mininet with connection to follower node1.
Start / End / Elapsed: 20250318 02:36:21.533 / 20250318 02:36:21.952 / 00:00:00.419
Status: FAIL
Message: Variable '${Follower_Node_2}' not found. Did you mean: ${Follower_Node_1}
00:01:41.035 TEST Verify Flows In Switch After Follower Node2 Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify Flows In Switch After Follower Node2 Restart
Documentation:

Verify flows are installed in switch after follower restart.
Start / End / Elapsed: 20250318 02:36:21.952 / 20250318 02:38:02.987 / 00:01:41.035
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: Non-existing index or alias '387'.
00:00:00.391 TEST Stop Mininet Connected To Follower Node2
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Stop Mininet Connected To Follower Node2
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250318 02:38:02.988 / 20250318 02:38:03.379 / 00:00:00.391
Status: FAIL
Message: Non-existing index or alias '387'.
00:00:00.365 TEST Delete All Flows From Follower Node 2
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Delete All Flows From Follower Node 2
Documentation:

1000 Flows deleted via Leader Node and verify it gets applied in all instances.
Start / End / Elapsed: 20250318 02:38:03.379 / 20250318 02:38:03.744 / 00:00:00.365
Status: FAIL
Message: Variable '${Follower_Node_2}' not found. Did you mean: ${Follower_Node_1}
00:00:00.468 TEST Verify No Flows In Cluster After Follower Node2 Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch.Verify No Flows In Cluster After Follower Node2 Restart
Start / End / Elapsed: 20250318 02:38:03.744 / 20250318 02:38:04.212 / 00:00:00.468
Status: FAIL
Message: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.21:8181/rests/operations/sal-bulk-flow:read-flow-test
00:00:08.056 SUITE Cluster Current Term Verification 3Node Cluster
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Current Term Verification 3Node Cluster
Documentation:

Test suite for OF-18 Spurious Leader Election verification of term change while flows are added/deleted
Source: /w/workspace/openflowplugin-csit-3node-clustering-bulkomatic-only-titanium/test/csit/suites/openflowplugin/Clustering_Bulkomatic/040__Cluster_Current_Term_Verification_3Node_Cluster.robot
Start / End / Elapsed: 20250318 02:38:04.214 / 20250318 02:38:12.270 / 00:00:08.056
Status: 12 tests total, 0 passed, 12 failed, 0 skipped