Test Statistics

Total Statistics
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
All Tests
1034162001:05:24
Statistics by Tag
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
No Tags
Statistics by Suite
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
openflowplugin-clustering-bulkomatic.txt
1034162001:05:40
openflowplugin-clustering-bulkomatic . txt . Cluster Reconcilliation Multi DPN
1899000:07:21
openflowplugin-clustering-bulkomatic . txt . Cluster HA Data Recovery BulkFlow 2Node Cluster
301119000:26:58
openflowplugin-clustering-bulkomatic . txt . Cluster HA Data Recovery BulkFlow Single Switch
432122000:31:12
openflowplugin-clustering-bulkomatic . txt . Cluster Current Term Verification 3Node Cluster
12012000:00:08

Test Execution Log

01:05:39.660 SUITE openflowplugin-clustering-bulkomatic.txt
Full Name: openflowplugin-clustering-bulkomatic.txt
Source: /w/workspace/openflowplugin-csit-3node-clustering-bulkomatic-only-titanium/test/csit/suites/openflowplugin/Clustering_Bulkomatic
Start / End / Elapsed: 20250128 01:35:36.914 / 20250128 02:41:16.574 / 01:05:39.660
Status: 103 tests total, 41 passed, 62 failed, 0 skipped
00:07:21.440 SUITE Cluster Reconcilliation Multi DPN
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN
Documentation:

Test suite for Cluster with Bulk Flows - Reconcilliation in a multi DPN environment
Source: /w/workspace/openflowplugin-csit-3node-clustering-bulkomatic-only-titanium/test/csit/suites/openflowplugin/Clustering_Bulkomatic/010__Cluster_Reconcilliation_Multi_DPN.robot
Start / End / Elapsed: 20250128 01:35:36.936 / 20250128 01:42:58.376 / 00:07:21.440
Status: 18 tests total, 9 passed, 9 failed, 0 skipped
00:00:05.102 SETUP SetupUtils . Setup_Utils_For_Setup_And_Teardown  
Documentation:

Prepare both FailFast and Karaf logging, log that the caller suite has started. This keyword is to be used in suite setup, and without it the other keywords may not work properly.
Start / End / Elapsed: 20250128 01:35:37.451 / 20250128 01:35:42.553 / 00:00:05.102
00:00:00.000 TEARDOWN RequestsLibrary . Delete All Sessions  
Documentation:

Removes all the session objects
Start / End / Elapsed: 20250128 01:42:58.376 / 20250128 01:42:58.376 / 00:00:00.000
00:00:00.530 TEST Check Shards Status And Initialize Variables
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Check Shards Status And Initialize Variables
Documentation:

Check Status for all shards in OpenFlow application.
Start / End / Elapsed: 20250128 01:35:42.553 / 20250128 01:35:43.083 / 00:00:00.530
Status: FAIL
Message: Dictionary '${Stream_dict}' has no key 'titanium'.
00:00:00.507 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250128 01:35:42.553 / 20250128 01:35:43.060 / 00:00:00.507
00:00:00.020 KEYWORD ClusterOpenFlow . Check OpenFlow Shards Status  
Documentation:

Check Status for all shards in OpenFlow application.
Start / End / Elapsed: 20250128 01:35:43.060 / 20250128 01:35:43.080 / 00:00:00.020
00:00:00.010 KEYWORD CompareStream . Run_Keyword_If_At_Least_Phosphorus   Collections.Remove Values From List ${SHARD_OPER_LIST} entity-ownership
Documentation:

Compare phosphorus to titanium and in case titanium is at least phosphorus, run ${kw_name} @{varargs} &{kwargs} and return its value.
Start / End / Elapsed: 20250128 01:35:43.071 / 20250128 01:35:43.081 / 00:00:00.010
00:00:00.009 KEYWORD BuiltIn . Run Keyword And Return   Run_Keyword_If_At_Least phosphorus ${kw_name} @{varargs} &{kwargs}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250128 01:35:43.071 / 20250128 01:35:43.080 / 00:00:00.009
00:00:00.001 KEYWORD CompareStream . Run_Keyword_If_At_Least   phosphorus ${kw_name} @{varargs} &{kwargs}
Documentation:

Compare ${lower_bound} to titanium and in case titanium is at least ${lower_bound}, run Collections.Remove Values From List [['inventory', 'topology', 'default', 'entity-ownership'], 'entity-ownership'] {} and return its value.
Start / End / Elapsed: 20250128 01:35:43.079 / 20250128 01:35:43.080 / 00:00:00.001
00:00:00.000 KEYWORD BuiltIn . Run Keyword And Return If   ${Stream_dict}[${ODL_STREAM}] >= ${Stream_dict}[${lower_bound}] ${kw_name} @{varargs} &{kwargs}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250128 01:35:43.079 / 20250128 01:35:43.079 / 00:00:00.000
01:35:43.080 FAIL Dictionary '${Stream_dict}' has no key 'titanium'.
01:35:43.080 INFO Returning from the enclosing user keyword.
00:00:00.000 KEYWORD BuiltIn . Log   ${SHARD_OPER_LIST}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250128 01:35:43.080 / 20250128 01:35:43.080 / 00:00:00.000
00:00:00.000 KEYWORD ClusterManagement . Verify_Leader_Exists_For_Each_Shard   shard_name_list=${SHARD_OPER_LIST} shard_type=operational member_index_list=${controller_index_list}
Documentation:

For each shard name, call Get_Leader_And_Followers_For_Shard. Not much logic there, but single Keyword is useful when using BuiltIn.Wait_Until_Keyword_Succeeds.
Start / End / Elapsed: 20250128 01:35:43.081 / 20250128 01:35:43.081 / 00:00:00.000
00:00:00.000 KEYWORD ClusterManagement . Verify_Leader_Exists_For_Each_Shard   shard_name_list=${SHARD_CONF_LIST} shard_type=config member_index_list=${controller_index_list}
Documentation:

For each shard name, call Get_Leader_And_Followers_For_Shard. Not much logic there, but single Keyword is useful when using BuiltIn.Wait_Until_Keyword_Succeeds.
Start / End / Elapsed: 20250128 01:35:43.081 / 20250128 01:35:43.081 / 00:00:00.000
00:00:00.000 KEYWORD ${temp_json_config_add} = BulkomaticKeywords . Set DPN And Flow Count In Json Add   ${orig_json_config_add} ${switch_count} ${flow_count_per_switch}
Documentation:

Set new DPN count and flows count per DPN in the Bulkomatic Add json file.
Start / End / Elapsed: 20250128 01:35:43.081 / 20250128 01:35:43.081 / 00:00:00.000
00:00:00.000 KEYWORD ${temp_json_config_get} = BulkomaticKeywords . Set DPN And Flow Count In Json Get   ${orig_json_config_get} ${switch_count} ${flow_count_after_add}
Documentation:

Set new DPN count and flows count per DPN in the Bulkomatic Get json file.
Start / End / Elapsed: 20250128 01:35:43.082 / 20250128 01:35:43.082 / 00:00:00.000
00:00:00.000 KEYWORD ${temp_json_config_del} = BulkomaticKeywords . Set DPN And Flow Count In Json Del   ${orig_json_config_del} ${switch_count} ${flow_count_per_switch}
Documentation:

Set new DPN count and flows count per DPN in the Bulkomatic Del json file.
Start / End / Elapsed: 20250128 01:35:43.082 / 20250128 01:35:43.082 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Set Suite Variable   ${temp_json_config_add}
Documentation:

Makes a variable available everywhere within the scope of the current suite.
Start / End / Elapsed: 20250128 01:35:43.082 / 20250128 01:35:43.082 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Set Suite Variable   ${temp_json_config_get}
Documentation:

Makes a variable available everywhere within the scope of the current suite.
Start / End / Elapsed: 20250128 01:35:43.082 / 20250128 01:35:43.082 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Set Suite Variable   ${temp_json_config_del}
Documentation:

Makes a variable available everywhere within the scope of the current suite.
Start / End / Elapsed: 20250128 01:35:43.082 / 20250128 01:35:43.082 / 00:00:00.000
00:00:06.452 TEST Get Inventory Follower and Leader Before Cluster Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Get Inventory Follower and Leader Before Cluster Restart
Documentation:

Find a follower in the inventory config shard
Start / End / Elapsed: 20250128 01:35:43.083 / 20250128 01:35:49.535 / 00:00:06.452
Status: PASS
00:00:03.427 TEST Start Mininet Connect To Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Start Mininet Connect To Follower Node1
Documentation:

Start mininet with connection to Follower Node1.
Start / End / Elapsed: 20250128 01:35:49.535 / 20250128 01:35:52.962 / 00:00:03.427
Status: PASS
00:00:00.387 TEST Add Bulk Flow From Follower
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Add Bulk Flow From Follower
Documentation:

10000 Flows ( 10000 per DPN) in 1 DPN added via Follower Node1 and verify it gets applied in all instances.
Start / End / Elapsed: 20250128 01:35:52.963 / 20250128 01:35:53.350 / 00:00:00.387
Status: FAIL
Message: Variable '${temp_json_config_add}' not found.
00:00:00.378 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250128 01:35:52.964 / 20250128 01:35:53.342 / 00:00:00.378
00:00:00.004 KEYWORD BulkomaticKeywords . Add Bulk Flow In Node   ${temp_json_config_add} ${Follower_Node_1} ${operation_timeout}
Documentation:

Add Bulk Flow in member ${controller_index} and wait until operation is completed.
Start / End / Elapsed: 20250128 01:35:53.346 / 20250128 01:35:53.350 / 00:00:00.004
01:35:53.350 FAIL Variable '${temp_json_config_add}' not found.
00:00:00.459 TEST Get Bulk Flows and Verify In Inventory Leader
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Get Bulk Flows and Verify In Inventory Leader
Documentation:

Initiate get operation and check flow count across cluster nodes
Start / End / Elapsed: 20250128 01:35:53.350 / 20250128 01:35:53.809 / 00:00:00.459
Status: FAIL
Message: Variable '${temp_json_config_get}' not found.
00:00:00.447 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250128 01:35:53.351 / 20250128 01:35:53.798 / 00:00:00.447
00:00:00.004 KEYWORD BulkomaticKeywords . Get Bulk Flow And Verify Count In Cluster   ${temp_json_config_get} ${operation_timeout} ${flow_count_after_add} ${Inventory_Leader_List}
Documentation:

Get Bulk Flow and Verify Flow Count in ${controller_index_list} matches ${flow_count}.
Start / End / Elapsed: 20250128 01:35:53.805 / 20250128 01:35:53.809 / 00:00:00.004
01:35:53.809 FAIL Variable '${temp_json_config_get}' not found.
00:01:41.555 TEST Verify Flows In Switch Before Cluster Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Verify Flows In Switch Before Cluster Restart
Documentation:

Verify flows are installed in switch before cluster restart.
Start / End / Elapsed: 20250128 01:35:53.809 / 20250128 01:37:35.364 / 00:01:41.555
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:00.389 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250128 01:35:53.810 / 20250128 01:35:54.199 / 00:00:00.389
00:01:41.164 KEYWORD MininetKeywords . Verify Aggregate Flow From Mininet Session   ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
Documentation:

Verify flow count per switch
Start / End / Elapsed: 20250128 01:35:54.199 / 20250128 01:37:35.363 / 00:01:41.164
00:01:41.164 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${time_out} 2s MininetKeywords.Check Flows In Mininet ${mininet_conn} ${flow_count}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250128 01:35:54.200 / 20250128 01:37:35.364 / 00:01:41.164
Message: 49 failing items removed using the --remove-keywords option.
00:00:00.036 KEYWORD MininetKeywords . Check Flows In Mininet   ${mininet_conn} ${flow_count}
Documentation:

Sync with mininet to match exact number of flows
Start / End / Elapsed: 20250128 01:37:35.327 / 20250128 01:37:35.363 / 00:00:00.036
00:00:00.001 IF """${mininet_conn}""" != ""  
Start / End / Elapsed: 20250128 01:37:35.328 / 20250128 01:37:35.329 / 00:00:00.001
00:00:00.000 KEYWORD ${cmd} = BuiltIn . Set Variable   dpctl dump-aggregate -O OpenFlow13
Documentation:

Returns the given values which can then be assigned to a variables.
Start / End / Elapsed: 20250128 01:37:35.330 / 20250128 01:37:35.330 / 00:00:00.000
00:00:00.030 KEYWORD ${output} = MininetKeywords . Send Mininet Command   ${mininet_conn} ${cmd}
Documentation:

Sends Command dpctl dump-aggregate -O OpenFlow13 to Mininet session 32 and returns read buffer response.
Start / End / Elapsed: 20250128 01:37:35.331 / 20250128 01:37:35.361 / 00:00:00.030
00:00:00.000 KEYWORD ${flows} = String . Get Regexp Matches   ${output} (?<=flow_count\=).*?(?=\r)
Documentation:

Returns a list of all non-overlapping matches in the given string.
Start / End / Elapsed: 20250128 01:37:35.361 / 20250128 01:37:35.361 / 00:00:00.000
00:00:00.000 KEYWORD ${total_flows} = BuiltIn . Evaluate   sum(map(int, ${flows}))
Documentation:

Evaluates the given expression in Python and returns the result.
Start / End / Elapsed: 20250128 01:37:35.362 / 20250128 01:37:35.362 / 00:00:00.000
00:00:00.001 KEYWORD BuiltIn . Should Be Equal As Numbers   ${total_flows} ${flow_count}
Documentation:

Fails if objects are unequal after converting them to real numbers.
Start / End / Elapsed: 20250128 01:37:35.363 / 20250128 01:37:35.364 / 00:00:00.001
01:37:35.363 INFO Argument types are: <class 'int'> <class 'str'>
01:37:35.363 FAIL 0.0 != 10000.0
01:37:35.364 FAIL Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:02.735 TEST Stop Mininet Connected To Follower Node1 and Exit
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Stop Mininet Connected To Follower Node1 and Exit
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250128 01:37:35.365 / 20250128 01:37:38.100 / 00:00:02.735
Status: PASS
00:00:02.853 TEST Start Mininet Reconnect To Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Start Mininet Reconnect To Follower Node1
Documentation:

Start mininet with reconnection to follower node1.
Start / End / Elapsed: 20250128 01:37:38.100 / 20250128 01:37:40.953 / 00:00:02.853
Status: PASS
00:01:40.418 TEST Verify Flows In Switch Reconnected To Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Verify Flows In Switch Reconnected To Follower Node1
Documentation:

Verify 10000 flows per DPN installed in switch after it is reconnected to follower node1.
Start / End / Elapsed: 20250128 01:37:40.954 / 20250128 01:39:21.372 / 00:01:40.418
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:00.393 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250128 01:37:40.954 / 20250128 01:37:41.347 / 00:00:00.393
00:01:40.024 KEYWORD MininetKeywords . Verify Aggregate Flow From Mininet Session   ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
Documentation:

Verify flow count per switch
Start / End / Elapsed: 20250128 01:37:41.348 / 20250128 01:39:21.372 / 00:01:40.024
00:01:40.023 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${time_out} 2s MininetKeywords.Check Flows In Mininet ${mininet_conn} ${flow_count}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250128 01:37:41.348 / 20250128 01:39:21.371 / 00:01:40.023
Message: 49 failing items removed using the --remove-keywords option.
00:00:00.037 KEYWORD MininetKeywords . Check Flows In Mininet   ${mininet_conn} ${flow_count}
Documentation:

Sync with mininet to match exact number of flows
Start / End / Elapsed: 20250128 01:39:21.335 / 20250128 01:39:21.372 / 00:00:00.037
00:00:00.001 IF """${mininet_conn}""" != ""  
Start / End / Elapsed: 20250128 01:39:21.336 / 20250128 01:39:21.337 / 00:00:00.001
00:00:00.000 KEYWORD ${cmd} = BuiltIn . Set Variable   dpctl dump-aggregate -O OpenFlow13
Documentation:

Returns the given values which can then be assigned to a variables.
Start / End / Elapsed: 20250128 01:39:21.337 / 20250128 01:39:21.337 / 00:00:00.000
00:00:00.031 KEYWORD ${output} = MininetKeywords . Send Mininet Command   ${mininet_conn} ${cmd}
Documentation:

Sends Command dpctl dump-aggregate -O OpenFlow13 to Mininet session 44 and returns read buffer response.
Start / End / Elapsed: 20250128 01:39:21.338 / 20250128 01:39:21.369 / 00:00:00.031
00:00:00.000 KEYWORD ${flows} = String . Get Regexp Matches   ${output} (?<=flow_count\=).*?(?=\r)
Documentation:

Returns a list of all non-overlapping matches in the given string.
Start / End / Elapsed: 20250128 01:39:21.370 / 20250128 01:39:21.370 / 00:00:00.000
00:00:00.001 KEYWORD ${total_flows} = BuiltIn . Evaluate   sum(map(int, ${flows}))
Documentation:

Evaluates the given expression in Python and returns the result.
Start / End / Elapsed: 20250128 01:39:21.370 / 20250128 01:39:21.371 / 00:00:00.001
00:00:00.000 KEYWORD BuiltIn . Should Be Equal As Numbers   ${total_flows} ${flow_count}
Documentation:

Fails if objects are unequal after converting them to real numbers.
Start / End / Elapsed: 20250128 01:39:21.371 / 20250128 01:39:21.371 / 00:00:00.000
01:39:21.371 INFO Argument types are: <class 'int'> <class 'str'>
01:39:21.371 FAIL 0.0 != 10000.0
01:39:21.371 FAIL Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:02.625 TEST Stop Mininet Connected To Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Stop Mininet Connected To Follower Node1
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250128 01:39:21.372 / 20250128 01:39:23.997 / 00:00:02.625
Status: PASS
00:00:02.750 TEST Start Mininet Connect To Follower Node2
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Start Mininet Connect To Follower Node2
Documentation:

Start mininet with connection to follower node2.
Start / End / Elapsed: 20250128 01:39:23.997 / 20250128 01:39:26.747 / 00:00:02.750
Status: PASS
00:01:40.403 TEST Verify Flows In Switch Connected To Follower Node2
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Verify Flows In Switch Connected To Follower Node2
Documentation:

Verify 10000 flows per DPN installed in switch after it is connected to follower node2.
Start / End / Elapsed: 20250128 01:39:26.747 / 20250128 01:41:07.150 / 00:01:40.403
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:00.377 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250128 01:39:26.748 / 20250128 01:39:27.125 / 00:00:00.377
00:01:40.025 KEYWORD MininetKeywords . Verify Aggregate Flow From Mininet Session   ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
Documentation:

Verify flow count per switch
Start / End / Elapsed: 20250128 01:39:27.125 / 20250128 01:41:07.150 / 00:01:40.025
00:01:40.025 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${time_out} 2s MininetKeywords.Check Flows In Mininet ${mininet_conn} ${flow_count}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250128 01:39:27.126 / 20250128 01:41:07.151 / 00:01:40.025
Message: 49 failing items removed using the --remove-keywords option.
00:00:00.042 KEYWORD MininetKeywords . Check Flows In Mininet   ${mininet_conn} ${flow_count}
Documentation:

Sync with mininet to match exact number of flows
Start / End / Elapsed: 20250128 01:41:07.108 / 20250128 01:41:07.150 / 00:00:00.042
00:00:00.001 IF """${mininet_conn}""" != ""  
Start / End / Elapsed: 20250128 01:41:07.109 / 20250128 01:41:07.110 / 00:00:00.001
00:00:00.000 KEYWORD ${cmd} = BuiltIn . Set Variable   dpctl dump-aggregate -O OpenFlow13
Documentation:

Returns the given values which can then be assigned to a variables.
Start / End / Elapsed: 20250128 01:41:07.110 / 20250128 01:41:07.110 / 00:00:00.000
00:00:00.038 KEYWORD ${output} = MininetKeywords . Send Mininet Command   ${mininet_conn} ${cmd}
Documentation:

Sends Command dpctl dump-aggregate -O OpenFlow13 to Mininet session 56 and returns read buffer response.
Start / End / Elapsed: 20250128 01:41:07.111 / 20250128 01:41:07.149 / 00:00:00.038
00:00:00.000 KEYWORD ${flows} = String . Get Regexp Matches   ${output} (?<=flow_count\=).*?(?=\r)
Documentation:

Returns a list of all non-overlapping matches in the given string.
Start / End / Elapsed: 20250128 01:41:07.149 / 20250128 01:41:07.149 / 00:00:00.000
00:00:00.000 KEYWORD ${total_flows} = BuiltIn . Evaluate   sum(map(int, ${flows}))
Documentation:

Evaluates the given expression in Python and returns the result.
Start / End / Elapsed: 20250128 01:41:07.149 / 20250128 01:41:07.149 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Should Be Equal As Numbers   ${total_flows} ${flow_count}
Documentation:

Fails if objects are unequal after converting them to real numbers.
Start / End / Elapsed: 20250128 01:41:07.150 / 20250128 01:41:07.150 / 00:00:00.000
01:41:07.150 INFO Argument types are: <class 'int'> <class 'str'>
01:41:07.150 FAIL 0.0 != 10000.0
01:41:07.150 FAIL Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:02.699 TEST Stop Mininet Connected To Follower Node2
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Stop Mininet Connected To Follower Node2
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250128 01:41:07.151 / 20250128 01:41:09.850 / 00:00:02.699
Status: PASS
00:00:02.695 TEST Start Mininet Connect To Inventory Leader
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Start Mininet Connect To Inventory Leader
Documentation:

Start mininet with connection to inventroy leader.
Start / End / Elapsed: 20250128 01:41:09.850 / 20250128 01:41:12.545 / 00:00:02.695
Status: PASS
00:01:42.335 TEST Verify Flows In Switch Connected To Leader
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Verify Flows In Switch Connected To Leader
Documentation:

Verify 10000 flows per DPN installed in switch after it is connected to inventory leader.
Start / End / Elapsed: 20250128 01:41:12.545 / 20250128 01:42:54.880 / 00:01:42.335
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:00.342 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250128 01:41:12.546 / 20250128 01:41:12.888 / 00:00:00.342
00:01:41.992 KEYWORD MininetKeywords . Verify Aggregate Flow From Mininet Session   ${mininet_conn_id} ${flow_count_after_add} ${operation_timeout}
Documentation:

Verify flow count per switch
Start / End / Elapsed: 20250128 01:41:12.888 / 20250128 01:42:54.880 / 00:01:41.992
00:01:41.991 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${time_out} 2s MininetKeywords.Check Flows In Mininet ${mininet_conn} ${flow_count}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250128 01:41:12.889 / 20250128 01:42:54.880 / 00:01:41.991
Message: 50 failing items removed using the --remove-keywords option.
00:00:00.033 KEYWORD MininetKeywords . Check Flows In Mininet   ${mininet_conn} ${flow_count}
Documentation:

Sync with mininet to match exact number of flows
Start / End / Elapsed: 20250128 01:42:54.846 / 20250128 01:42:54.879 / 00:00:00.033
00:00:00.001 IF """${mininet_conn}""" != ""  
Start / End / Elapsed: 20250128 01:42:54.847 / 20250128 01:42:54.848 / 00:00:00.001
00:00:00.000 KEYWORD ${cmd} = BuiltIn . Set Variable   dpctl dump-aggregate -O OpenFlow13
Documentation:

Returns the given values which can then be assigned to a variables.
Start / End / Elapsed: 20250128 01:42:54.849 / 20250128 01:42:54.849 / 00:00:00.000
00:00:00.028 KEYWORD ${output} = MininetKeywords . Send Mininet Command   ${mininet_conn} ${cmd}
Documentation:

Sends Command dpctl dump-aggregate -O OpenFlow13 to Mininet session 68 and returns read buffer response.
Start / End / Elapsed: 20250128 01:42:54.850 / 20250128 01:42:54.878 / 00:00:00.028
00:00:00.000 KEYWORD ${flows} = String . Get Regexp Matches   ${output} (?<=flow_count\=).*?(?=\r)
Documentation:

Returns a list of all non-overlapping matches in the given string.
Start / End / Elapsed: 20250128 01:42:54.878 / 20250128 01:42:54.878 / 00:00:00.000
00:00:00.000 KEYWORD ${total_flows} = BuiltIn . Evaluate   sum(map(int, ${flows}))
Documentation:

Evaluates the given expression in Python and returns the result.
Start / End / Elapsed: 20250128 01:42:54.879 / 20250128 01:42:54.879 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Should Be Equal As Numbers   ${total_flows} ${flow_count}
Documentation:

Fails if objects are unequal after converting them to real numbers.
Start / End / Elapsed: 20250128 01:42:54.879 / 20250128 01:42:54.879 / 00:00:00.000
01:42:54.879 INFO Argument types are: <class 'int'> <class 'str'>
01:42:54.879 FAIL 0.0 != 10000.0
01:42:54.880 FAIL Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:02.765 TEST Stop Mininet Connected To Inventory Leader
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Stop Mininet Connected To Inventory Leader
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250128 01:42:54.880 / 20250128 01:42:57.645 / 00:00:02.765
Status: PASS
00:00:00.378 TEST Delete All Flows From Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Delete All Flows From Follower Node1
Documentation:

10000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
Start / End / Elapsed: 20250128 01:42:57.645 / 20250128 01:42:58.023 / 00:00:00.378
Status: FAIL
Message: Variable '${temp_json_config_del}' not found.
00:00:00.369 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250128 01:42:57.646 / 20250128 01:42:58.015 / 00:00:00.369
00:00:00.004 KEYWORD BulkomaticKeywords . Delete Bulk Flow In Node   ${temp_json_config_del} ${Follower_Node_1} ${operation_timeout}
Documentation:

Delete Bulk Flow in member ${controller_index} and wait until operation is completed.
Start / End / Elapsed: 20250128 01:42:58.019 / 20250128 01:42:58.023 / 00:00:00.004
01:42:58.023 FAIL Variable '${temp_json_config_del}' not found.
00:00:00.352 TEST Verify No Flows In Inventory Leader
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Reconcilliation Multi DPN.Verify No Flows In Inventory Leader
Documentation:

Verify flow count is 0 across cluster nodes.
Start / End / Elapsed: 20250128 01:42:58.024 / 20250128 01:42:58.376 / 00:00:00.352
Status: FAIL
Message: Variable '${temp_json_config_get}' not found.
00:00:00.339 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250128 01:42:58.024 / 20250128 01:42:58.363 / 00:00:00.339
00:00:00.004 KEYWORD BulkomaticKeywords . Get Bulk Flow And Verify Count In Cluster   ${temp_json_config_get} ${operation_timeout} ${flow_count_after_del} ${Inventory_Leader_List}
Documentation:

Get Bulk Flow and Verify Flow Count in ${controller_index_list} matches ${flow_count}.
Start / End / Elapsed: 20250128 01:42:58.372 / 20250128 01:42:58.376 / 00:00:00.004
01:42:58.375 FAIL Variable '${temp_json_config_get}' not found.
00:26:58.283 SUITE Cluster HA Data Recovery BulkFlow 2Node Cluster
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster
Documentation:

Test suite for 2Node Cluster HA with Bulk Flows - Cluster node convergance and Data consistency after leader and follower restart with one switch connected
Source: /w/workspace/openflowplugin-csit-3node-clustering-bulkomatic-only-titanium/test/csit/suites/openflowplugin/Clustering_Bulkomatic/020__Cluster_HA_Data_Recovery_BulkFlow_2Node_Cluster.robot
Start / End / Elapsed: 20250128 01:42:58.377 / 20250128 02:09:56.660 / 00:26:58.283
Status: 30 tests total, 11 passed, 19 failed, 0 skipped
00:00:03.821 SETUP SetupUtils . Setup_Utils_For_Setup_And_Teardown  
Documentation:

Prepare both FailFast and Karaf logging, log that the caller suite has started. This keyword is to be used in suite setup, and without it the other keywords may not work properly.
Start / End / Elapsed: 20250128 01:42:58.444 / 20250128 01:43:02.265 / 00:00:03.821
00:00:00.000 TEARDOWN RequestsLibrary . Delete All Sessions  
Documentation:

Removes all the session objects
Start / End / Elapsed: 20250128 02:09:56.660 / 20250128 02:09:56.660 / 00:00:00.000
00:00:00.428 TEST Check Shards Status and Initialize Variables
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Check Shards Status and Initialize Variables
Documentation:

Check Status for all shards in OpenFlow application.
Start / End / Elapsed: 20250128 01:43:02.265 / 20250128 01:43:02.693 / 00:00:00.428
Status: FAIL
Message: Dictionary '${Stream_dict}' has no key 'titanium'.
00:00:00.406 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250128 01:43:02.265 / 20250128 01:43:02.671 / 00:00:00.406
00:00:00.019 KEYWORD ClusterOpenFlow . Check OpenFlow Shards Status  
Documentation:

Check Status for all shards in OpenFlow application.
Start / End / Elapsed: 20250128 01:43:02.671 / 20250128 01:43:02.690 / 00:00:00.019
00:00:00.009 KEYWORD CompareStream . Run_Keyword_If_At_Least_Phosphorus   Collections.Remove Values From List ${SHARD_OPER_LIST} entity-ownership
Documentation:

Compare phosphorus to titanium and in case titanium is at least phosphorus, run ${kw_name} @{varargs} &{kwargs} and return its value.
Start / End / Elapsed: 20250128 01:43:02.681 / 20250128 01:43:02.690 / 00:00:00.009
00:00:00.008 KEYWORD BuiltIn . Run Keyword And Return   Run_Keyword_If_At_Least phosphorus ${kw_name} @{varargs} &{kwargs}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250128 01:43:02.681 / 20250128 01:43:02.689 / 00:00:00.008
00:00:00.001 KEYWORD CompareStream . Run_Keyword_If_At_Least   phosphorus ${kw_name} @{varargs} &{kwargs}
Documentation:

Compare ${lower_bound} to titanium and in case titanium is at least ${lower_bound}, run Collections.Remove Values From List [['inventory', 'topology', 'default', 'entity-ownership'], 'entity-ownership'] {} and return its value.
Start / End / Elapsed: 20250128 01:43:02.689 / 20250128 01:43:02.690 / 00:00:00.001
00:00:00.000 KEYWORD BuiltIn . Run Keyword And Return If   ${Stream_dict}[${ODL_STREAM}] >= ${Stream_dict}[${lower_bound}] ${kw_name} @{varargs} &{kwargs}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250128 01:43:02.690 / 20250128 01:43:02.690 / 00:00:00.000
01:43:02.690 INFO Returning from the enclosing user keyword.
00:00:00.000 KEYWORD BuiltIn . Log   ${SHARD_OPER_LIST}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250128 01:43:02.690 / 20250128 01:43:02.690 / 00:00:00.000
00:00:00.000 KEYWORD ClusterManagement . Verify_Leader_Exists_For_Each_Shard   shard_name_list=${SHARD_OPER_LIST} shard_type=operational member_index_list=${controller_index_list}
Documentation:

For each shard name, call Get_Leader_And_Followers_For_Shard. Not much logic there, but single Keyword is useful when using BuiltIn.Wait_Until_Keyword_Succeeds.
Start / End / Elapsed: 20250128 01:43:02.690 / 20250128 01:43:02.690 / 00:00:00.000
00:00:00.000 KEYWORD ClusterManagement . Verify_Leader_Exists_For_Each_Shard   shard_name_list=${SHARD_CONF_LIST} shard_type=config member_index_list=${controller_index_list}
Documentation:

For each shard name, call Get_Leader_And_Followers_For_Shard. Not much logic there, but single Keyword is useful when using BuiltIn.Wait_Until_Keyword_Succeeds.
Start / End / Elapsed: 20250128 01:43:02.691 / 20250128 01:43:02.691 / 00:00:00.000
00:00:00.000 KEYWORD ${temp_json_config_add} = BulkomaticKeywords . Set DPN And Flow Count In Json Add   ${orig_json_config_add} ${switch_count} ${flow_count_per_switch}
Documentation:

Set new DPN count and flows count per DPN in the Bulkomatic Add json file.
Start / End / Elapsed: 20250128 01:43:02.691 / 20250128 01:43:02.691 / 00:00:00.000
00:00:00.000 KEYWORD ${temp_json_config_get} = BulkomaticKeywords . Set DPN And Flow Count In Json Get   ${orig_json_config_get} ${switch_count} ${flow_count_after_add}
Documentation:

Set new DPN count and flows count per DPN in the Bulkomatic Get json file.
Start / End / Elapsed: 20250128 01:43:02.691 / 20250128 01:43:02.691 / 00:00:00.000
00:00:00.000 KEYWORD ${temp_json_config_del} = BulkomaticKeywords . Set DPN And Flow Count In Json Del   ${orig_json_config_del} ${switch_count} ${flow_count_per_switch}
Documentation:

Set new DPN count and flows count per DPN in the Bulkomatic Del json file.
Start / End / Elapsed: 20250128 01:43:02.692 / 20250128 01:43:02.692 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Set Suite Variable   ${temp_json_config_add}
Documentation:

Makes a variable available everywhere within the scope of the current suite.
Start / End / Elapsed: 20250128 01:43:02.692 / 20250128 01:43:02.692 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Set Suite Variable   ${temp_json_config_get}
Documentation:

Makes a variable available everywhere within the scope of the current suite.
Start / End / Elapsed: 20250128 01:43:02.692 / 20250128 01:43:02.692 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Set Suite Variable   ${temp_json_config_del}
Documentation:

Makes a variable available everywhere within the scope of the current suite.
Start / End / Elapsed: 20250128 01:43:02.692 / 20250128 01:43:02.692 / 00:00:00.000
00:00:01.132 TEST Get Inventory Follower Before Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Get Inventory Follower Before Leader Restart
Documentation:

Find a follower in the inventory config shard
Start / End / Elapsed: 20250128 01:43:02.693 / 20250128 01:43:03.825 / 00:00:01.132
Status: PASS
00:00:03.713 TEST Shutdown Leader From Cluster Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Shutdown Leader From Cluster Node
Documentation:

Shutdown Leader Node and Start it Up
Start / End / Elapsed: 20250128 01:43:03.825 / 20250128 01:43:07.538 / 00:00:03.713
Status: PASS
00:01:51.617 TEST Check Shards Status After Leader Shutdown
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Check Shards Status After Leader Shutdown
Documentation:

Wait for node convergence and check status for all shards in OpenFlow application.
Start / End / Elapsed: 20250128 01:43:07.539 / 20250128 01:44:59.156 / 00:01:51.617
Status: FAIL
Message: Keyword 'ClusterOpenFlow.Check OpenFlow Shards Status' failed after retrying for 1 minute 40 seconds. The last error was: Dictionary '${Stream_dict}' has no key 'titanium'.
00:00:00.828 TEST Check Shard Status For Leader After PreLeader Shutdown
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Check Shard Status For Leader After PreLeader Shutdown
Documentation:

Find a Leader in the inventory config shard
Start / End / Elapsed: 20250128 01:44:59.156 / 20250128 01:44:59.984 / 00:00:00.828
Status: PASS
00:00:03.076 TEST Start Mininet Connect To Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Start Mininet Connect To Follower Node1
Documentation:

Start mininet with connection to Follower Node1
Start / End / Elapsed: 20250128 01:44:59.985 / 20250128 01:45:03.061 / 00:00:03.076
Status: PASS
00:00:00.303 TEST Add Bulk Flow From Follower
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Add Bulk Flow From Follower
Documentation:

10000 Flows added via Follower Node1 and verify it gets applied in all instances.
Start / End / Elapsed: 20250128 01:45:03.061 / 20250128 01:45:03.364 / 00:00:00.303
Status: FAIL
Message: Variable '${temp_json_config_add}' not found.
00:00:00.304 TEST Get Bulk Flows And Verify In Leader
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Get Bulk Flows And Verify In Leader
Documentation:

Initiate get operation and check flow count across cluster nodes.
Start / End / Elapsed: 20250128 01:45:03.364 / 20250128 01:45:03.668 / 00:00:00.304
Status: FAIL
Message: Variable '${temp_json_config_get}' not found.
00:01:42.230 TEST Verify Flows In Switch Before Cluster Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Verify Flows In Switch Before Cluster Restart
Documentation:

Verify flows are installed in switch before cluster restart.
Start / End / Elapsed: 20250128 01:45:03.668 / 20250128 01:46:45.898 / 00:01:42.230
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:25.185 TEST Restart Pre Leader From Cluster Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Restart Pre Leader From Cluster Node
Documentation:

Restart Leader Node.
Start / End / Elapsed: 20250128 01:46:45.899 / 20250128 01:47:11.084 / 00:00:25.185
Status: PASS
00:05:51.905 TEST Check Shards Status After Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Check Shards Status After Leader Restart
Documentation:

Wait for node convergence and check status for all shards in OpenFlow application.
Start / End / Elapsed: 20250128 01:47:11.085 / 20250128 01:53:02.990 / 00:05:51.905
Status: FAIL
Message: Keyword 'ClusterOpenFlow.Check OpenFlow Shards Status' failed after retrying for 5 minutes 50 seconds. The last error was: Dictionary '${Stream_dict}' has no key 'titanium'.
00:01:41.385 TEST Verify Data Recovery After Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Verify Data Recovery After Leader Restart
Documentation:

10000 Flows preserved in all controller instances.
Start / End / Elapsed: 20250128 01:53:02.990 / 20250128 01:54:44.375 / 00:01:41.385
Status: FAIL
Message: Keyword 'BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster' failed after retrying for 1 minute 40 seconds. The last error was: Variable '${temp_json_config_get}' not found.
00:01:42.422 TEST Verify Flows In Switch After Leader Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Verify Flows In Switch After Leader Restart
Documentation:

Verify flows are installed in switch after cluster restart.
Start / End / Elapsed: 20250128 01:54:44.375 / 20250128 01:56:26.797 / 00:01:42.422
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:02.635 TEST Stop Mininet Connected To Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Stop Mininet Connected To Follower Node1
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250128 01:56:26.797 / 20250128 01:56:29.432 / 00:00:02.635
Status: PASS
00:00:00.397 TEST Delete All Flows From Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Delete All Flows From Follower Node1
Documentation:

10000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
Start / End / Elapsed: 20250128 01:56:29.433 / 20250128 01:56:29.830 / 00:00:00.397
Status: FAIL
Message: Variable '${temp_json_config_del}' not found.
00:00:00.368 TEST Verify No Flows In Leader Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Verify No Flows In Leader Node
Documentation:

Verify flow count is 0 across cluster nodes.
Start / End / Elapsed: 20250128 01:56:29.830 / 20250128 01:56:30.198 / 00:00:00.368
Status: FAIL
Message: Variable '${temp_json_config_get}' not found.
00:00:01.214 TEST Get Inventory Follower And Leader Before Cluster Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Get Inventory Follower And Leader Before Cluster Restart
Documentation:

Find a follower in the inventory config shard
Start / End / Elapsed: 20250128 01:56:30.198 / 20250128 01:56:31.412 / 00:00:01.214
Status: PASS
00:00:04.459 TEST Shutdown Follower From Cluster Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Shutdown Follower From Cluster Node
Documentation:

Shutdown Follower Node2 and Start it Up.
Start / End / Elapsed: 20250128 01:56:31.412 / 20250128 01:56:35.871 / 00:00:04.459
Status: PASS
00:01:51.679 TEST Check Shards Status After Follower Shutdown
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Check Shards Status After Follower Shutdown
Documentation:

Wait for node convergence and check status for all shards in OpenFlow application.
Start / End / Elapsed: 20250128 01:56:35.871 / 20250128 01:58:27.550 / 00:01:51.679
Status: FAIL
Message: Keyword 'ClusterOpenFlow.Check OpenFlow Shards Status' failed after retrying for 1 minute 40 seconds. The last error was: Dictionary '${Stream_dict}' has no key 'titanium'.
00:00:03.016 TEST Start Mininet Connect To Follower Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Start Mininet Connect To Follower Node
Documentation:

Start mininet with connection to Follower Node1.
Start / End / Elapsed: 20250128 01:58:27.550 / 20250128 01:58:30.566 / 00:00:03.016
Status: PASS
00:00:00.286 TEST Add Bulk Flow From Follower Node1
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Add Bulk Flow From Follower Node1
Documentation:

10000 Flows added via Follower Node1 and verify it gets applied in all instances.
Start / End / Elapsed: 20250128 01:58:30.567 / 20250128 01:58:30.853 / 00:00:00.286
Status: FAIL
Message: Variable '${temp_json_config_add}' not found.
00:00:00.383 TEST Get Bulk Flows And Verify In Leader Before Follower Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Get Bulk Flows And Verify In Leader Before Follower Restart
Documentation:

Initiate get operation and check flow count 10000 only across active cluster nodes
Start / End / Elapsed: 20250128 01:58:30.853 / 20250128 01:58:31.236 / 00:00:00.383
Status: FAIL
Message: Variable '${temp_json_config_get}' not found.
00:01:40.358 TEST Verify Flows In Switch Before Follower Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Verify Flows In Switch Before Follower Restart
Documentation:

Verify flows are installed in switch before follower restart.
Start / End / Elapsed: 20250128 01:58:31.235 / 20250128 02:00:11.593 / 00:01:40.358
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:25.273 TEST Restart Follower From Cluster Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Restart Follower From Cluster Node
Documentation:

Restart Follower Node2.
Start / End / Elapsed: 20250128 02:00:11.594 / 20250128 02:00:36.867 / 00:00:25.273
Status: PASS
00:05:52.130 TEST Check Shards Status After Follower Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Check Shards Status After Follower Restart
Documentation:

Wait for node convergence and check status for all shards in OpenFlow application.
Start / End / Elapsed: 20250128 02:00:36.866 / 20250128 02:06:28.996 / 00:05:52.130
Status: FAIL
Message: Keyword 'ClusterOpenFlow.Check OpenFlow Shards Status' failed after retrying for 5 minutes 50 seconds. The last error was: Dictionary '${Stream_dict}' has no key 'titanium'.
00:01:41.485 TEST Verify Data Recovery After Follower Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Verify Data Recovery After Follower Restart
Documentation:

10000 Flows preserved in all controller instances.
Start / End / Elapsed: 20250128 02:06:28.997 / 20250128 02:08:10.482 / 00:01:41.485
Status: FAIL
Message: Keyword 'BulkomaticKeywords.Get Bulk Flow And Verify Count In Cluster' failed after retrying for 1 minute 40 seconds. The last error was: Variable '${temp_json_config_get}' not found.
00:01:42.498 TEST Verify Flows In Switch After Follower Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Verify Flows In Switch After Follower Restart
Documentation:

Verify flows are installed in switch after cluster restart.
Start / End / Elapsed: 20250128 02:08:10.482 / 20250128 02:09:52.980 / 00:01:42.498
Status: FAIL
Message: Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 0.0 != 10000.0
00:00:02.866 TEST Stop Mininet Connected To Follower Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Stop Mininet Connected To Follower Node
Documentation:

Stop mininet and exit connection.
Start / End / Elapsed: 20250128 02:09:52.980 / 20250128 02:09:55.846 / 00:00:02.866
Status: PASS
00:00:00.385 TEST Delete All Flows From Follower Node
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Delete All Flows From Follower Node
Documentation:

10000 Flows deleted via Follower Node1 and verify it gets applied in all instances.
Start / End / Elapsed: 20250128 02:09:55.847 / 20250128 02:09:56.232 / 00:00:00.385
Status: FAIL
Message: Variable '${temp_json_config_del}' not found.
00:00:00.427 TEST Verify No Flows In Leader Node After Follower Restart
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow 2Node Cluster.Verify No Flows In Leader Node After Follower Restart
Documentation:

Verify flow count is 0 across cluster nodes.
Start / End / Elapsed: 20250128 02:09:56.232 / 20250128 02:09:56.659 / 00:00:00.427
Status: FAIL
Message: Variable '${temp_json_config_get}' not found.
00:31:11.666 SUITE Cluster HA Data Recovery BulkFlow Single Switch
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster HA Data Recovery BulkFlow Single Switch
Documentation:

Test suite for Cluster HA with Bulk Flows - Data consistency after cluster restart, leader restart and follower restart with one switch connected
Source: /w/workspace/openflowplugin-csit-3node-clustering-bulkomatic-only-titanium/test/csit/suites/openflowplugin/Clustering_Bulkomatic/030__Cluster_HA_Data_Recovery_BulkFlow_Single_Switch.robot
Start / End / Elapsed: 20250128 02:09:56.661 / 20250128 02:41:08.327 / 00:31:11.666
Status: 43 tests total, 21 passed, 22 failed, 0 skipped
00:00:08.244 SUITE Cluster Current Term Verification 3Node Cluster
Full Name: openflowplugin-clustering-bulkomatic.txt.Cluster Current Term Verification 3Node Cluster
Documentation:

Test suite for OF-18 Spurious Leader Election verification of term change while flows are added/deleted
Source: /w/workspace/openflowplugin-csit-3node-clustering-bulkomatic-only-titanium/test/csit/suites/openflowplugin/Clustering_Bulkomatic/040__Cluster_Current_Term_Verification_3Node_Cluster.robot
Start / End / Elapsed: 20250128 02:41:08.328 / 20250128 02:41:16.572 / 00:00:08.244
Status: 12 tests total, 0 passed, 12 failed, 0 skipped