Test Statistics

Total Statistics
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
All Tests
19522173003:30:56
Statistics by Tag
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
carpeople
85085000:00:00
clustering
85085000:00:00
critical
19522173003:30:56
Statistics by Suite
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
controller-clustering-ask.txt
19522173003:56:24
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:41
controller-clustering-ask . txt . Rpc Provider Precedence
844000:14:41
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:21
controller-clustering-ask . txt . Rpc Provider Partition And Heal
707000:09:18
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:14
controller-clustering-ask . txt . Action Provider Precedence
844000:14:41
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:23
controller-clustering-ask . txt . Action Provider Partition And Heal
725000:14:01
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:26
controller-clustering-ask . txt . Master Stability
606000:09:08
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:26
controller-clustering-ask . txt . Partition And Heal
606000:09:08
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:21
controller-clustering-ask . txt . Chasing The Leader
312000:03:55
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:21
controller-clustering-ask . txt . Global Rpc Kill
11011000:01:03
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:20
controller-clustering-ask . txt . Global Rpc Freeze
10010000:01:04
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:21
controller-clustering-ask . txt . Global Rpc Isolate
11011000:01:04
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:22
controller-clustering-ask . txt . Carpeople Crud
13013000:00:54
controller-clustering-ask . txt . Car Failover Crud
16016000:00:53
controller-clustering-ask . txt . Car Persistence Recovery
707000:00:53
controller-clustering-ask . txt . Buycar Failover
10010000:00:53
controller-clustering-ask . txt . Entity Isolate
505000:00:54
controller-clustering-ask . txt . Buycar Failover Isolation
10010000:00:53
controller-clustering-ask . txt . Car Failover Crud Isolation
16016000:00:53
controller-clustering-ask . txt . Car Outage Corners
808000:00:53

Test Execution Log

03:56:24.264 SUITE controller-clustering-ask.txt
Full Name: controller-clustering-ask.txt
Start / End / Elapsed: 20250316 23:08:34.161 / 20250317 03:04:58.425 / 03:56:24.264
Status: 195 tests total, 22 passed, 173 failed, 0 skipped
00:13:40.618 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250316 23:08:34.184 / 20250316 23:22:14.802 / 00:13:40.618
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:04.810 SETUP SetupUtils . Setup_Utils_For_Setup_And_Teardown   http_timeout=125
Documentation:

Prepare both FailFast and Karaf logging, log that the caller suite has started. This keyword is to be used in suite setup, and without it the other keywords may not work properly.
Start / End / Elapsed: 20250316 23:08:34.738 / 20250316 23:08:39.548 / 00:00:04.810
00:00:00.000 TEARDOWN SSHLibrary . Close All Connections  
Documentation:

Closes all open connections.
Start / End / Elapsed: 20250316 23:22:14.802 / 20250316 23:22:14.802 / 00:00:00.000
00:06:13.902 TEST Stop_All_Members
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Stop_All_Members
Documentation:

Stop every odl node.
Tags: critical
Start / End / Elapsed: 20250316 23:08:39.548 / 20250316 23:14:53.450 / 00:06:13.902
Status: FAIL
Message: Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1
00:06:13.901 KEYWORD ClusterManagement . Stop_Members_From_List_Or_All  
Documentation:

If the list is empty, stops all ODL instances. Otherwise stop members based on ${stop_index_list} If ${confirm} is True, verify stopped instances are not there anymore. The KW will return a list of available members: ${updated index_list}=${original_index_list}-${member_index_list}
Start / End / Elapsed: 20250316 23:08:39.549 / 20250316 23:14:53.450 / 00:06:13.901
00:00:00.001 KEYWORD ${stop_index_list} = ClusterManagement . List_Indices_Or_All   given_list=${member_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250316 23:08:39.553 / 20250316 23:08:39.554 / 00:00:00.001
00:00:00.001 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All   given_list=${original_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250316 23:08:39.554 / 20250316 23:08:39.555 / 00:00:00.001
00:00:02.488 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=${NODE_STOP_COMMAND} member_index_list=${member_index_list}
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250316 23:08:39.556 / 20250316 23:08:42.044 / 00:00:02.488
00:00:00.000 KEYWORD ${updated_index_list} = BuiltIn . Create List   @{index_list}
Documentation:

Returns a list containing given items.
Start / End / Elapsed: 20250316 23:08:42.044 / 20250316 23:08:42.044 / 00:00:00.000
00:00:00.000 KEYWORD Collections . Remove Values From List   ${updated_index_list} @{stop_index_list}
Documentation:

Removes all occurrences of given values from list.
Start / End / Elapsed: 20250316 23:08:42.045 / 20250316 23:08:42.045 / 00:00:00.000
00:00:00.000 IF not ${confirm}  
Start / End / Elapsed: 20250316 23:08:42.045 / 20250316 23:08:42.045 / 00:00:00.000
00:06:11.405 FOR ${index} IN @{stop_index_list}  
Start / End / Elapsed: 20250316 23:08:42.045 / 20250316 23:14:53.450 / 00:06:11.405
00:00:07.920 ITERATION ${index} = 1  
Start / End / Elapsed: 20250316 23:08:42.045 / 20250316 23:08:49.965 / 00:00:07.920
00:00:00.458 ITERATION ${index} = 2  
Start / End / Elapsed: 20250316 23:08:49.965 / 20250316 23:08:50.423 / 00:00:00.458
00:06:03.027 ITERATION ${index} = 3  
Start / End / Elapsed: 20250316 23:08:50.423 / 20250316 23:14:53.450 / 00:06:03.027
00:06:03.027 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${timeout} 2s Verify_Karaf_Is_Not_Running_On_Member member_index=${index}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250316 23:08:50.423 / 20250316 23:14:53.450 / 00:06:03.027
Message: 145 failing items removed using the --remove-keywords option.
00:00:01.203 KEYWORD ClusterManagement . Verify_Karaf_Is_Not_Running_On_Member   member_index=${index}
Documentation:

Fail if non-zero karaf instances are counted on member of given index.
Start / End / Elapsed: 20250316 23:14:52.247 / 20250316 23:14:53.450 / 00:00:01.203
00:00:01.201 KEYWORD ${count} = ClusterManagement . Count_Running_Karafs_On_Member   member_index=${member_index}
Documentation:

Remotely execute grep for karaf process, return count as string.
Start / End / Elapsed: 20250316 23:14:52.248 / 20250316 23:14:53.449 / 00:00:01.201
00:00:00.000 KEYWORD BuiltIn . Should Be Equal   0 ${count} Found running Karaf count: ${count}
Documentation:

Fails if the given objects are unequal.
Start / End / Elapsed: 20250316 23:14:53.449 / 20250316 23:14:53.449 / 00:00:00.000
23:14:53.450 FAIL Found running Karaf count: 1: 0 != 1
23:14:53.450 FAIL Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1
00:00:00.000 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=netstat -pnatu | grep 2550
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250316 23:14:53.450 / 20250316 23:14:53.450 / 00:00:00.000
00:00:00.000 RETURN   ${updated_index_list}
Start / End / Elapsed: 20250316 23:14:53.450 / 20250316 23:14:53.450 / 00:00:00.000
00:00:10.976 TEST Unset_Tell_Based_Protocol_Usage
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Unset_Tell_Based_Protocol_Usage
Documentation:

Comment out the flag usage in config file. Also clean most data except data/log/.
Tags: critical
Start / End / Elapsed: 20250316 23:14:53.451 / 20250316 23:15:04.427 / 00:00:10.976
Status: PASS
00:07:10.370 TEST Start_All_And_Sync
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Start_All_And_Sync
Documentation:

Start each member and wait for sync.
Tags: critical
Start / End / Elapsed: 20250316 23:15:04.431 / 20250316 23:22:14.801 / 00:07:10.370
Status: FAIL
Message: Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: ReadTimeout: HTTPConnectionPool(host='10.30.171.207', port=8181): Read timed out. (read timeout=125.0)
00:07:10.363 KEYWORD ClusterManagement . Start_Members_From_List_Or_All  
Documentation:

If the list is empty, start all cluster members. Otherwise, start members based on present indices. If ${wait_for_sync}, wait for cluster sync on listed members. Optionally karaf_home can be overriden. Optionally specific JAVA_HOME is used for starting. Garbage collection is unconditionally logged to files. TODO: Make that reasonable conditional?
Start / End / Elapsed: 20250316 23:15:04.436 / 20250316 23:22:14.799 / 00:07:10.363
00:00:00.000 KEYWORD ${base_command} = BuiltIn . Set Variable If   """${karaf_home}""" != "" ${karaf_home}/bin/start ${NODE_START_COMMAND}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250316 23:15:04.443 / 20250316 23:15:04.443 / 00:00:00.000
00:00:00.000 KEYWORD ${command} = BuiltIn . Set Variable If   """${export_java_home}""" != "" export JAVA_HOME="${export_java_home}"; ${base_command} ${base_command}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250316 23:15:04.444 / 20250316 23:15:04.444 / 00:00:00.000
00:00:00.000 KEYWORD ${epoch} = DateTime . Get Current Date   time_zone=UTC result_format=epoch exclude_millis=False
Documentation:

Returns current local or UTC time with an optional increment.
Start / End / Elapsed: 20250316 23:15:04.444 / 20250316 23:15:04.444 / 00:00:00.000
00:00:00.000 KEYWORD ${gc_filepath} = BuiltIn . Set Variable If   """${karaf_home}""" != "" ${karaf_home}/data/log/gc_${epoch}.log ${GC_LOG_PATH}/gc_${epoch}.log
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250316 23:15:04.445 / 20250316 23:15:04.445 / 00:00:00.000
00:00:00.000 KEYWORD ${gc_options} = BuiltIn . Set Variable If   "docker" not in """${node_start_command}""" -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:${gc_filepath} ${EMPTY}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250316 23:15:04.445 / 20250316 23:15:04.445 / 00:00:00.000
00:00:01.450 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=${command} ${gc_options} member_index_list=${member_index_list}
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250316 23:15:04.446 / 20250316 23:15:05.896 / 00:00:01.450
00:07:06.896 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${timeout} 10s Verify_Members_Are_Ready ${member_index_list} ${wait_for_sync} ${verify_restconf} ${check_system_status} ${service_list}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250316 23:15:05.896 / 20250316 23:22:12.792 / 00:07:06.896
Message: 5 failing items removed using the --remove-keywords option.
00:02:05.280 KEYWORD ClusterManagement . Verify_Members_Are_Ready   ${member_index_list} ${wait_for_sync} ${verify_restconf} ${check_system_status} ${service_list}
Documentation:

Verifies the specified readiness conditions for the given listed members after startup. If ${verify_cluster_sync}, verifies the datastores have synced with the rest of the cluster. If True, verifies RESTCONF is available. If ${verify_system_status}, verifies the system services are OPERATIONAL.
Start / End / Elapsed: 20250316 23:20:07.511 / 20250316 23:22:12.791 / 00:02:05.280
00:02:05.278 IF ${verify_cluster_sync}  
Start / End / Elapsed: 20250316 23:20:07.512 / 20250316 23:22:12.790 / 00:02:05.278
00:02:05.278 KEYWORD ClusterManagement . Check_Cluster_Is_In_Sync   ${member_index_list}
Documentation:

Fail if no-sync is detected on a member from list (or any).
Start / End / Elapsed: 20250316 23:20:07.513 / 20250316 23:22:12.791 / 00:02:05.278
00:00:00.002 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All   given_list=${member_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250316 23:20:07.513 / 20250316 23:20:07.515 / 00:00:00.002
00:02:05.275 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250316 23:20:07.515 / 20250316 23:22:12.790 / 00:02:05.275
00:00:00.089 ITERATION ${index} = 1  
Start / End / Elapsed: 20250316 23:20:07.516 / 20250316 23:20:07.605 / 00:00:00.089
00:00:00.076 ITERATION ${index} = 2  
Start / End / Elapsed: 20250316 23:20:07.606 / 20250316 23:20:07.682 / 00:00:00.076
00:02:05.108 ITERATION ${index} = 3  
Start / End / Elapsed: 20250316 23:20:07.683 / 20250316 23:22:12.791 / 00:02:05.108
00:02:05.107 KEYWORD ${status} = ClusterManagement . Get_Sync_Status_Of_Member   member_index=${index}
Documentation:

Obtain IP, two GETs from jolokia URIs, return combined sync status as string.
Start / End / Elapsed: 20250316 23:20:07.683 / 20250316 23:22:12.790 / 00:02:05.107
00:00:00.001 KEYWORD ${session} = ClusterManagement . Resolve_Http_Session_For_Member   member_index=${member_index}
Documentation:

Return RequestsLibrary session alias pointing to node of given index.
Start / End / Elapsed: 20250316 23:20:07.683 / 20250316 23:20:07.684 / 00:00:00.001
00:02:05.105 KEYWORD ${conf_text} = TemplatedRequests . Get_As_Json_From_Uri   uri=${JOLOKIA_CONF_SHARD_MANAGER_URI} session=${session}
Documentation:

Specify JSON headers and return Get_From_Uri normalized response text.
Start / End / Elapsed: 20250316 23:20:07.684 / 20250316 23:22:12.789 / 00:02:05.105
00:02:05.102 KEYWORD ${response_text} = TemplatedRequests . Get_From_Uri   uri=${uri} accept=${ACCEPT_EMPTY} session=${session} normalize_json=True http_timeout=${http_timeout} log_response=${log_response}
Documentation:

GET data from given URI, check status code and return response text. ${accept} is a Python object with headers to use. If ${normalize_json}, normalize as JSON text before returning.
Start / End / Elapsed: 20250316 23:20:07.687 / 20250316 23:22:12.789 / 00:02:05.102
00:00:00.000 KEYWORD BuiltIn . Log   ${uri}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250316 23:20:07.691 / 20250316 23:20:07.691 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${accept}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250316 23:20:07.691 / 20250316 23:20:07.691 / 00:00:00.000
00:02:05.093 IF """${http_timeout}""" == """${EMPTY}"""  
Start / End / Elapsed: 20250316 23:20:07.691 / 20250316 23:22:12.784 / 00:02:05.093
00:02:05.091 KEYWORD ${response} = RequestsLibrary . GET On Session   ${session} url=${uri} headers=${accept}
Documentation:

Sends a GET request on a previously created HTTP Session.
Start / End / Elapsed: 20250316 23:20:07.694 / 20250316 23:22:12.785 / 00:02:05.091
23:22:12.781 FAIL ReadTimeout: HTTPConnectionPool(host='10.30.171.207', port=8181): Read timed out. (read timeout=125.0)
00:00:00.001 ELSE  
Start / End / Elapsed: 20250316 23:22:12.785 / 20250316 23:22:12.786 / 00:00:00.001
00:00:00.000 KEYWORD TemplatedRequests . Check_Status_Code   ${response} log_response=${log_response}
Documentation:

Log response text, check status_code is one of allowed ones. In cases where this keyword is called in a WUKS it could end up logging tons of data and it may be desired to skip the logging by passing log_response=False, but by default it remains True.
Start / End / Elapsed: 20250316 23:22:12.787 / 20250316 23:22:12.787 / 00:00:00.000
00:00:00.000 IF not ${normalize_json}  
Start / End / Elapsed: 20250316 23:22:12.787 / 20250316 23:22:12.787 / 00:00:00.000
00:00:00.000 KEYWORD ${text_normalized} = norm_json . Normalize Json Text   ${response.text} jmes_path=${jmes_path} keys_with_volatiles=${keys_with_volatiles}
Documentation:

Attempt to return sorted indented JSON string.
Start / End / Elapsed: 20250316 23:22:12.788 / 20250316 23:22:12.788 / 00:00:00.000
00:00:00.000 RETURN   ${text_normalized}
Start / End / Elapsed: 20250316 23:22:12.788 / 20250316 23:22:12.788 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250316 23:22:12.788 / 20250316 23:22:12.788 / 00:00:00.000
00:00:00.000 KEYWORD ${conf_status} = ClusterManagement . ClusterManagement__Parse_Sync_Status   shard_manager_text=${conf_text}
Documentation:

Return sync status parsed out of given text. Called twice by Get_Sync_Status_Of_Member.
Start / End / Elapsed: 20250316 23:22:12.789 / 20250316 23:22:12.789 / 00:00:00.000
00:00:00.000 IF 'False' == ${conf_status}  
Start / End / Elapsed: 20250316 23:22:12.789 / 20250316 23:22:12.789 / 00:00:00.000
00:00:00.000 KEYWORD ${oper_text} = TemplatedRequests . Get_As_Json_From_Uri   uri=${JOLOKIA_OPER_SHARD_MANAGER_URI} session=${session}
Documentation:

Specify JSON headers and return Get_From_Uri normalized response text.
Start / End / Elapsed: 20250316 23:22:12.790 / 20250316 23:22:12.790 / 00:00:00.000
00:00:00.000 KEYWORD ${oper_status} = ClusterManagement . ClusterManagement__Parse_Sync_Status   shard_manager_text=${oper_text}
Documentation:

Return sync status parsed out of given text. Called twice by Get_Sync_Status_Of_Member.
Start / End / Elapsed: 20250316 23:22:12.790 / 20250316 23:22:12.790 / 00:00:00.000
00:00:00.000 RETURN   ${oper_status}
Start / End / Elapsed: 20250316 23:22:12.790 / 20250316 23:22:12.790 / 00:00:00.000
00:00:00.000 IF 'True' == '${status}'  
Start / End / Elapsed: 20250316 23:22:12.790 / 20250316 23:22:12.790 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Fail   Index ${index} has incorrect status: ${status}
Documentation:

Fails the test with the given message and optionally alters its tags.
Start / End / Elapsed: 20250316 23:22:12.790 / 20250316 23:22:12.790 / 00:00:00.000
00:00:00.000 IF ${verify_restconf}  
Start / End / Elapsed: 20250316 23:22:12.791 / 20250316 23:22:12.791 / 00:00:00.000
00:00:00.000 IF ${verify_system_status} and ("${service_list}" != "[[]]")  
Start / End / Elapsed: 20250316 23:22:12.791 / 20250316 23:22:12.791 / 00:00:00.000
23:22:12.792 FAIL Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: ReadTimeout: HTTPConnectionPool(host='10.30.171.207', port=8181): Read timed out. (read timeout=125.0)
00:00:02.007 TEARDOWN ClusterManagement . Run_Bash_Command_On_List_Or_All   command=netstat -pnatu | grep 2550
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250316 23:22:12.792 / 20250316 23:22:14.799 / 00:00:02.007
00:00:00.000 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 10s ClusterManagement.Run_Bash_Command_On_List_Or_All netstat -punta
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250316 23:22:14.799 / 20250316 23:22:14.799 / 00:00:00.000
00:00:00.000 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All  
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250316 23:22:14.800 / 20250316 23:22:14.800 / 00:00:00.000
00:00:00.001 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250316 23:22:14.800 / 20250316 23:22:14.801 / 00:00:00.001
00:00:00.000 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 10s ShardStability.Shards_Stability_Get_Details ${DEFAULT_SHARD_LIST} verify_restconf=True
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250316 23:22:14.801 / 20250316 23:22:14.801 / 00:00:00.000
00:14:40.683 SUITE Rpc Provider Precedence
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence
Documentation:

DOMRpcBroker testing: RPC Provider Precedence

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

The aim is to establish that remote RPC implementations have lower priority than local ones, which is to say that any movement of RPCs on remote nodes does not affect routing as long as a local implementation is available.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot
Start / End / Elapsed: 20250316 23:22:14.803 / 20250316 23:36:55.486 / 00:14:40.683
Status: 8 tests total, 4 passed, 4 failed, 0 skipped
00:00:53.711 SETUP Setup_Keyword  
Documentation:

Create a list of possible constant responses on the node with unregistered rpc.
Start / End / Elapsed: 20250316 23:22:14.887 / 20250316 23:23:08.598 / 00:00:53.711
00:00:00.000 TEARDOWN SSHLibrary . Close All Connections  
Documentation:

Closes all open connections.
Start / End / Elapsed: 20250316 23:36:55.485 / 20250316 23:36:55.485 / 00:00:00.000
00:02:47.213 TEST Register_Rpc_On_Each_Node
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence.Register_Rpc_On_Each_Node
Documentation:

Register global rpc on each node of the cluster.
Tags: critical
Start / End / Elapsed: 20250316 23:23:08.598 / 20250316 23:25:55.811 / 00:02:47.213
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_precedence_register_rpc_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_precedence_register_rpc_on_each_node&order=bug_status" ReadTimeout: HTTPConnectionPool(host='10.30.171.207', port=8181): Read timed out. (read timeout=125.0)
00:00:40.310 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250316 23:23:08.598 / 20250316 23:23:48.908 / 00:00:40.310
00:02:06.890 KEYWORD DrbCommons . Register_Rpc_On_Nodes   ${all_indices}
Documentation:

Register global rpc on given nodes of the cluster.
Start / End / Elapsed: 20250316 23:23:48.908 / 20250316 23:25:55.798 / 00:02:06.890
00:02:06.889 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250316 23:23:48.909 / 20250316 23:25:55.798 / 00:02:06.889
00:00:00.831 ITERATION ${index} = 1  
Start / End / Elapsed: 20250316 23:23:48.909 / 20250316 23:23:49.740 / 00:00:00.831
00:00:00.814 ITERATION ${index} = 2  
Start / End / Elapsed: 20250316 23:23:49.740 / 20250316 23:23:50.554 / 00:00:00.814
00:02:05.244 ITERATION ${index} = 3  
Start / End / Elapsed: 20250316 23:23:50.554 / 20250316 23:25:55.798 / 00:02:05.244
00:02:05.244 KEYWORD DrbCommons . Register_Rpc_And_Update_Possible_Constants   ${index}
Documentation:

Register global rpc on given node of the cluster.
Start / End / Elapsed: 20250316 23:23:50.554 / 20250316 23:25:55.798 / 00:02:05.244
00:02:05.243 KEYWORD MdsalLowlevel . Register_Constant   ${member_index} ${CONSTANT_PREFIX}${member_index}
Documentation:

Register the get-constant rpc on the requested node with given constant.
Start / End / Elapsed: 20250316 23:23:50.554 / 20250316 23:25:55.797 / 00:02:05.243
00:00:00.001 KEYWORD ${session} = ClusterManagement . Resolve_Http_Session_For_Member   member_index=${member_index}
Documentation:

Return RequestsLibrary session alias pointing to node of given index.
Start / End / Elapsed: 20250316 23:23:50.555 / 20250316 23:23:50.556 / 00:00:00.001
00:00:00.000 KEYWORD &{mapping} = BuiltIn . Create Dictionary   CONSTANT=${constant}
Documentation:

Creates and returns a dictionary based on the given items.
Start / End / Elapsed: 20250316 23:23:50.556 / 20250316 23:23:50.556 / 00:00:00.000
00:02:05.240 KEYWORD TemplatedRequests . Post_As_Xml_Templated   ${REGISTER_CONSTANT_DIR} mapping=${mapping} session=${session}
Documentation:

Add arguments sensible for XML data, return Post_Templated response text. Optionally, verification against response.xml (no iteration) is called.
Start / End / Elapsed: 20250316 23:23:50.556 / 20250316 23:25:55.796 / 00:02:05.240
00:02:05.235 KEYWORD ${response_text} = TemplatedRequests . Post_Templated   folder=${folder} base_name=data extension=xml accept=${ACCEPT_XML} content_type=${HEADERS_XML} mapping=${mapping} session=${session} normalize_json=False endline=${\n} iterations=${iterations} iter_start=${iter_start} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes} http_timeout=${http_timeout} iter_j_offset=${iter_j_offset}
Documentation:

Resolve URI and data from folder, call Post_To_Uri, return response text.
Start / End / Elapsed: 20250316 23:23:50.561 / 20250316 23:25:55.796 / 00:02:05.235
00:00:00.062 KEYWORD ${uri} = TemplatedRequests . Resolve_Text_From_Template_Folder   folder=${folder} base_name=location extension=uri mapping=${mapping} percent_encode=True
Documentation:

Read a template from folder, strip endline, make changes according to mapping, return the result. If ${iterations} value is present, put text together from "prolog", "item" and "epilog" parts, where additional template variable ${i} goes from 1, by one times. Template variable ${j} is calculated as ${i} incremented by offset 0 ( j = i + iter_j_offset ) used to create non uniform data in order to be able to validate UPDATE operations. POST (as opposed to PUT) needs slightly different data, ${name_prefix} may be used to distinguish. (Actually, it is GET who formats data differently when URI is a top-level container.)
Start / End / Elapsed: 20250316 23:23:50.571 / 20250316 23:23:50.633 / 00:00:00.062
00:00:00.052 KEYWORD ${data} = TemplatedRequests . Resolve_Text_From_Template_Folder   folder=${folder} name_prefix=post_ base_name=${base_name} extension=${extension} mapping=${mapping} endline=${endline} iterations=${iterations} iter_start=${iter_start} iter_j_offset=${iter_j_offset}
Documentation:

Read a template from folder, strip endline, make changes according to mapping, return the result. If ${iterations} value is present, put text together from "prolog", "item" and "epilog" parts, where additional template variable ${i} goes from 1, by one times. Template variable ${j} is calculated as ${i} incremented by offset 0 ( j = i + iter_j_offset ) used to create non uniform data in order to be able to validate UPDATE operations. POST (as opposed to PUT) needs slightly different data, ${name_prefix} may be used to distinguish. (Actually, it is GET who formats data differently when URI is a top-level container.)
Start / End / Elapsed: 20250316 23:23:50.642 / 20250316 23:23:50.694 / 00:00:00.052
00:00:00.004 KEYWORD ${jmes_expression} = TemplatedRequests . Resolve_Jmes_Path   ${folder}
Documentation:

Reads JMES path from file /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/libraries/../variables/mdsal/lowlevelrpc/register_constant/jmespath.expr if the file exists and returns the JMES path. Empty string is returned otherwise.
Start / End / Elapsed: 20250316 23:23:50.694 / 20250316 23:23:50.698 / 00:00:00.004
00:02:05.097 KEYWORD ${response_text} = TemplatedRequests . Post_To_Uri   uri=${uri} data=${data} content_type=${content_type} accept=${accept} session=${session} jmes_path=${jmes_expression} normalize_json=${normalize_json} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes} http_timeout=${http_timeout}
Documentation:

POST data to given URI, check status code and return response text. ${content_type} and ${accept} are mandatory Python objects with headers to use. If ${normalize_json}, normalize text before returning.
Start / End / Elapsed: 20250316 23:23:50.699 / 20250316 23:25:55.796 / 00:02:05.097
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250316 23:25:55.796 / 20250316 23:25:55.796 / 00:00:00.000
00:00:00.001 IF ${verify}  
Start / End / Elapsed: 20250316 23:25:55.796 / 20250316 23:25:55.797 / 00:00:00.001
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250316 23:25:55.797 / 20250316 23:25:55.797 / 00:00:00.000
00:00:00.000 KEYWORD DrbCommons . DrbCommons__Add_Possible_Constant   ${member_index}
Documentation:

Add a constant to the ['constant-1', 'constant-2'] list. The list is about to maintain all valid constants possibly responded from the odl cluster (excluding isolated nodes).
Start / End / Elapsed: 20250316 23:25:55.797 / 20250316 23:25:55.797 / 00:00:00.000
00:00:00.000 KEYWORD DrbCommons . DrbCommons__Register_Index   ${member_index}
Documentation:

Add member index to the list of indices with registered rpc. Isolated nodes are included in the list.
Start / End / Elapsed: 20250316 23:25:55.798 / 20250316 23:25:55.798 / 00:00:00.000
00:00:00.012 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs in case of failure.
Start / End / Elapsed: 20250316 23:25:55.799 / 20250316 23:25:55.811 / 00:00:00.012
00:02:46.027 TEST Invoke_Rpc_On_Each_Node
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence.Invoke_Rpc_On_Each_Node
Documentation:

Verify that the rpc response comes from the local node.
Tags: critical
Start / End / Elapsed: 20250316 23:25:55.811 / 20250316 23:28:41.838 / 00:02:46.027
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_precedence_invoke_rpc_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_precedence_invoke_rpc_on_each_node&order=bug_status" ReadTimeout: HTTPConnectionPool(host='10.30.171.207', port=8181): Read timed out. (read timeout=125.0)
00:00:40.481 TEST Unregister_Rpc_On_Node
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence.Unregister_Rpc_On_Node
Documentation:

Unregister the rpc on one of the cluster nodes.
Tags: critical
Start / End / Elapsed: 20250316 23:28:41.838 / 20250316 23:29:22.319 / 00:00:40.481
Status: PASS
00:00:40.509 TEST Invoke_Rpc_On_Node_With_Unregistered_Rpc
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence.Invoke_Rpc_On_Node_With_Unregistered_Rpc
Documentation:

Invoke rcp on the node with unregistered rpc. The response is expected to come from other nodes where the rpc remained registered.
Tags: critical
Start / End / Elapsed: 20250316 23:29:22.319 / 20250316 23:30:02.828 / 00:00:40.509
Status: PASS
00:00:40.455 TEST Invoke_Rpc_On_Remaining_Nodes
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence.Invoke_Rpc_On_Remaining_Nodes
Documentation:

Verify that the rpc response comes from the local node.
Tags: critical
Start / End / Elapsed: 20250316 23:30:02.828 / 20250316 23:30:43.283 / 00:00:40.455
Status: PASS
00:00:40.431 TEST Reregister_Rpc_On_Node
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence.Reregister_Rpc_On_Node
Documentation:

Reregister the rpc.
Tags: critical
Start / End / Elapsed: 20250316 23:30:43.283 / 20250316 23:31:23.714 / 00:00:40.431
Status: PASS
00:02:45.905 TEST Invoke_Rpc_On_Each_Node_Again
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence.Invoke_Rpc_On_Each_Node_Again
Documentation:

Verify that the rpc response comes from the local node.
Tags: critical
Start / End / Elapsed: 20250316 23:31:23.714 / 20250316 23:34:09.619 / 00:02:45.905
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_precedence_invoke_rpc_on_each_node_again" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_precedence_invoke_rpc_on_each_node_again&order=bug_status" ReadTimeout: HTTPConnectionPool(host='10.30.171.207', port=8181): Read timed out. (read timeout=125.0)
00:02:45.866 TEST Unregister_Rpc_On_Each_Node
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence.Unregister_Rpc_On_Each_Node
Documentation:

Unregister rpc on every node.
Tags: critical
Start / End / Elapsed: 20250316 23:34:09.619 / 20250316 23:36:55.485 / 00:02:45.866
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_precedence_unregister_rpc_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_precedence_unregister_rpc_on_each_node&order=bug_status" ReadTimeout: HTTPConnectionPool(host='10.30.171.207', port=8181): Read timed out. (read timeout=125.0)
00:13:20.945 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250316 23:36:55.486 / 20250316 23:50:16.431 / 00:13:20.945
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:09:17.593 SUITE Rpc Provider Partition And Heal
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal
Documentation:

DOMRpcBroker testing: RPC Provider Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This tests establishes that the RPC service operates correctly when faced with node failures. This suite supports more than three node cluster setup too.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot
Start / End / Elapsed: 20250316 23:50:16.433 / 20250316 23:59:34.026 / 00:09:17.593
Status: 7 tests total, 0 passed, 7 failed, 0 skipped
00:14:13.649 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250316 23:59:34.027 / 20250317 00:13:47.676 / 00:14:13.649
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:14:41.022 SUITE Action Provider Precedence
Full Name: controller-clustering-ask.txt.Action Provider Precedence
Documentation:

DOMRpcBroker testing: Action Provider Precedence

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

The aim is to establish that remote action implementations have lower priority than local ones, which is to say that any movement of actions on remote nodes does not affect routing as long as a local implementation is available.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot
Start / End / Elapsed: 20250317 00:13:47.676 / 20250317 00:28:28.698 / 00:14:41.022
Status: 8 tests total, 4 passed, 4 failed, 0 skipped
00:14:23.407 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250317 00:28:28.699 / 20250317 00:42:52.106 / 00:14:23.407
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:14:01.099 SUITE Action Provider Partition And Heal
Full Name: controller-clustering-ask.txt.Action Provider Partition And Heal
Documentation:

DOMRpcBroker testing: RPC Action Provider Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This tests establishes that the RPC service for actions operates correctly when faced with node failures. This suite supports more than three node cluster setup too.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot
Start / End / Elapsed: 20250317 00:42:52.107 / 20250317 00:56:53.206 / 00:14:01.099
Status: 7 tests total, 2 passed, 5 failed, 0 skipped
00:14:25.676 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250317 00:56:53.206 / 20250317 01:11:18.882 / 00:14:25.676
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:09:07.857 SUITE Master Stability
Full Name: controller-clustering-ask.txt.Master Stability
Documentation:

Cluster Singleton testing: Master Stability

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Cluster Singleton service is designed to ensure that only one instance of an application is registered globally in the cluster. The goal is to establish the service operates correctly in face of application registration changing without moving the active instance.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/cluster_singleton/master_stability.robot
Start / End / Elapsed: 20250317 01:11:18.883 / 20250317 01:20:26.740 / 00:09:07.857
Status: 6 tests total, 0 passed, 6 failed, 0 skipped
00:14:25.666 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250317 01:20:26.740 / 20250317 01:34:52.406 / 00:14:25.666
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:09:08.407 SUITE Partition And Heal
Full Name: controller-clustering-ask.txt.Partition And Heal
Documentation:

Cluster Singleton testing: Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Cluster Singleton service is designed to ensure that only one instance of an application is registered globally in the cluster. The goal is to establish the service operates correctly in face of node failures.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot
Start / End / Elapsed: 20250317 01:34:52.407 / 20250317 01:44:00.814 / 00:09:08.407
Status: 6 tests total, 0 passed, 6 failed, 0 skipped
00:13:20.912 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250317 01:44:00.815 / 20250317 01:57:21.727 / 00:13:20.912
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:03:55.337 SUITE Chasing The Leader
Full Name: controller-clustering-ask.txt.Chasing The Leader
Documentation:

Cluster Singleton testing: Chasing the Leader

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test aims to establish the service operates correctly when faced with rapid application transitions without having a stabilized application.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot
Start / End / Elapsed: 20250317 01:57:21.727 / 20250317 02:01:17.064 / 00:03:55.337
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:13:21.236 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250317 02:01:17.064 / 20250317 02:14:38.300 / 00:13:21.236
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:01:03.372 SUITE Global Rpc Kill
Full Name: controller-clustering-ask.txt.Global Rpc Kill
Documentation:

Controller functional HA testing of global singleton rpcs.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot
Start / End / Elapsed: 20250317 02:14:38.301 / 20250317 02:15:41.673 / 00:01:03.372
Status: 11 tests total, 0 passed, 11 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:20.289 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250317 02:15:41.674 / 20250317 02:29:01.963 / 00:13:20.289
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:01:03.629 SUITE Global Rpc Freeze
Full Name: controller-clustering-ask.txt.Global Rpc Freeze
Documentation:

Controller functional HA testing of global singleton rpcs if jvm frozen.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot
Start / End / Elapsed: 20250317 02:29:01.964 / 20250317 02:30:05.593 / 00:01:03.629
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:21.026 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250317 02:30:05.594 / 20250317 02:43:26.620 / 00:13:21.026
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:01:03.530 SUITE Global Rpc Isolate
Full Name: controller-clustering-ask.txt.Global Rpc Isolate
Documentation:

Controller functional HA testing of global singleton rpcs.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot
Start / End / Elapsed: 20250317 02:43:26.620 / 20250317 02:44:30.150 / 00:01:03.530
Status: 11 tests total, 0 passed, 11 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:22.305 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250317 02:44:30.151 / 20250317 02:57:52.456 / 00:13:22.305
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:54.051 SUITE Carpeople Crud
Full Name: controller-clustering-ask.txt.Carpeople Crud
Documentation:

Suite for performing basic car/people CRUD operations on leaders and followers.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

More precisely, Update operation is not executed, but some operations are using specific RPCs which goes beyond "basic CRUD".

Cars are added by one big PUT to datastore on car Leader. People are added in a loop with add-person RPC on a people Follower. Cars are bought by chunks on each member, by loop with buy-car RPC.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot
Start / End / Elapsed: 20250317 02:57:52.456 / 20250317 02:58:46.507 / 00:00:54.051
Status: 13 tests total, 0 passed, 13 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:52.754 SUITE Car Failover Crud
Full Name: controller-clustering-ask.txt.Car Failover Crud
Documentation:

Suite mixing basic operations with restart of car Leader.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test stops the current leader of the "car" shard and then executes CRD operations on the new leader and a new follower. The stopped member is brought back. This suite uses 3 different car sets, same size but different starting ID.

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot
Start / End / Elapsed: 20250317 02:58:46.507 / 20250317 02:59:39.261 / 00:00:52.754
Status: 16 tests total, 0 passed, 16 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:53.452 SUITE Car Persistence Recovery
Full Name: controller-clustering-ask.txt.Car Persistence Recovery
Documentation:

This test restarts all controllers to verify recovery of car data from persistence.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot
Start / End / Elapsed: 20250317 02:59:39.262 / 20250317 03:00:32.714 / 00:00:53.452
Status: 7 tests total, 0 passed, 7 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:52.561 SUITE Buycar Failover
Full Name: controller-clustering-ask.txt.Buycar Failover
Documentation:

This test focuses on testing buy-car RPC over 3 Leader reboots.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

All purchases are against the same node, which is the first one to get rebooted.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot
Start / End / Elapsed: 20250317 03:00:32.715 / 20250317 03:01:25.276 / 00:00:52.561
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:54.484 SUITE Entity Isolate
Full Name: controller-clustering-ask.txt.Entity Isolate
Documentation:

Suite for performing member isolation and rejoin, we do with entity-ownership leader.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot
Start / End / Elapsed: 20250317 03:01:25.276 / 20250317 03:02:19.760 / 00:00:54.484
Status: 5 tests total, 0 passed, 5 failed, 0 skipped
Message: Suite setup failed: Evaluating expression 'json.loads(\'\'\'{\n "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "error_type": "javax.management.InstanceNotFoundException",\n "request": {\n "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "type": "read"\n },\n "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1073)\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1343)\\n\\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:921)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\\n\\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\\n\\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:233)\\n\\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:245)\\n\\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:221)\\n\\tat org.jolokia.handler.ReadHandler.f... [ Message content over the limit has been removed. ] ...lipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\\n\\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\\n\\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\\n\\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\\n\\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\\n\\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\\n\\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\\n\\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\\n\\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\\n\\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\\n\\tat java.base/java.lang.Thread.run(Thread.java:1583)\\n",\n "status": 404\n}\n\'\'\')' failed: JSONDecodeError: Invalid control character at: line 8 column 190 (char 619)
00:00:52.579 SUITE Buycar Failover Isolation
Full Name: controller-clustering-ask.txt.Buycar Failover Isolation
Documentation:

This test focuses on testing buy-car RPC over the Leader isolation.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

All purchases are against the same node, which is the first one to get isolated.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot
Start / End / Elapsed: 20250317 03:02:19.761 / 20250317 03:03:12.340 / 00:00:52.579
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:53.385 SUITE Car Failover Crud Isolation
Full Name: controller-clustering-ask.txt.Car Failover Crud Isolation
Documentation:

Suite mixing basic operations with isolation of car Leader.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test isolates the current leader of the "car" shard and then executes CRD operations on the new leader and a new follower. The isolated member is brought back. This suite uses 3 different car sets, same size but different starting ID.

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot
Start / End / Elapsed: 20250317 03:03:12.341 / 20250317 03:04:05.726 / 00:00:53.385
Status: 16 tests total, 0 passed, 16 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:52.694 SUITE Car Outage Corners
Full Name: controller-clustering-ask.txt.Car Outage Corners
Documentation:

Cluster suite for testing minimal and sum-minimal member population behavior.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test stops majority of the followers and verifies car addition is not possible, then resumes single follower (first from original list) and checks that addition works. Then remaining members are brought up. Leader member is always up and assumed to remain Leading during the whole suite run.

TODO: Use initial data to check more operations. TODO: Perhaps merge with car_failover_crud suite.

Other modules and Shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot
Start / End / Elapsed: 20250317 03:04:05.727 / 20250317 03:04:58.421 / 00:00:52.694
Status: 8 tests total, 0 passed, 8 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig