Test Statistics

Total Statistics
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
All Tests
19519176003:06:08
Statistics by Tag
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
carpeople
85085000:00:00
clustering
85085000:00:00
critical
19519176003:06:08
Statistics by Suite
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
controller-clustering-ask.txt
19519176003:31:38
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:52
controller-clustering-ask . txt . Rpc Provider Precedence
844000:14:41
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:24
controller-clustering-ask . txt . Rpc Provider Partition And Heal
707000:09:20
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:24
controller-clustering-ask . txt . Action Provider Precedence
835000:06:19
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:24
controller-clustering-ask . txt . Action Provider Partition And Heal
707000:09:20
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:25
controller-clustering-ask . txt . Master Stability
606000:04:57
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:25
controller-clustering-ask . txt . Partition And Heal
606000:04:57
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:24
controller-clustering-ask . txt . Chasing The Leader
312000:03:56
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:21
controller-clustering-ask . txt . Global Rpc Kill
11011000:01:05
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:24
controller-clustering-ask . txt . Global Rpc Freeze
10010000:01:04
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:22
controller-clustering-ask . txt . Global Rpc Isolate
11011000:01:04
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:26
controller-clustering-ask . txt . Carpeople Crud
13013000:00:54
controller-clustering-ask . txt . Car Failover Crud
16016000:00:52
controller-clustering-ask . txt . Car Persistence Recovery
707000:00:53
controller-clustering-ask . txt . Buycar Failover
10010000:00:52
controller-clustering-ask . txt . Entity Isolate
505000:00:54
controller-clustering-ask . txt . Buycar Failover Isolation
10010000:00:52
controller-clustering-ask . txt . Car Failover Crud Isolation
16016000:00:53
controller-clustering-ask . txt . Car Outage Corners
808000:00:52

Test Execution Log

03:31:37.591 SUITE controller-clustering-ask.txt
Full Name: controller-clustering-ask.txt
Start / End / Elapsed: 20250302 22:58:18.233 / 20250303 02:29:55.824 / 03:31:37.591
Status: 195 tests total, 19 passed, 176 failed, 0 skipped
00:13:52.016 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250302 22:58:18.254 / 20250302 23:12:10.270 / 00:13:52.016
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:05.273 SETUP SetupUtils . Setup_Utils_For_Setup_And_Teardown   http_timeout=125
Documentation:

Prepare both FailFast and Karaf logging, log that the caller suite has started. This keyword is to be used in suite setup, and without it the other keywords may not work properly.
Start / End / Elapsed: 20250302 22:58:18.784 / 20250302 22:58:24.057 / 00:00:05.273
00:00:00.000 TEARDOWN SSHLibrary . Close All Connections  
Documentation:

Closes all open connections.
Start / End / Elapsed: 20250302 23:12:10.270 / 20250302 23:12:10.270 / 00:00:00.000
00:06:13.268 TEST Stop_All_Members
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Stop_All_Members
Documentation:

Stop every odl node.
Tags: critical
Start / End / Elapsed: 20250302 22:58:24.057 / 20250302 23:04:37.325 / 00:06:13.268
Status: FAIL
Message: Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1
00:06:13.267 KEYWORD ClusterManagement . Stop_Members_From_List_Or_All  
Documentation:

If the list is empty, stops all ODL instances. Otherwise stop members based on ${stop_index_list} If ${confirm} is True, verify stopped instances are not there anymore. The KW will return a list of available members: ${updated index_list}=${original_index_list}-${member_index_list}
Start / End / Elapsed: 20250302 22:58:24.058 / 20250302 23:04:37.325 / 00:06:13.267
00:00:00.001 KEYWORD ${stop_index_list} = ClusterManagement . List_Indices_Or_All   given_list=${member_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250302 22:58:24.062 / 20250302 22:58:24.063 / 00:00:00.001
00:00:00.001 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All   given_list=${original_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250302 22:58:24.063 / 20250302 22:58:24.064 / 00:00:00.001
00:00:02.504 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=${NODE_STOP_COMMAND} member_index_list=${member_index_list}
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250302 22:58:24.065 / 20250302 22:58:26.569 / 00:00:02.504
00:00:00.000 KEYWORD ${updated_index_list} = BuiltIn . Create List   @{index_list}
Documentation:

Returns a list containing given items.
Start / End / Elapsed: 20250302 22:58:26.569 / 20250302 22:58:26.569 / 00:00:00.000
00:00:00.000 KEYWORD Collections . Remove Values From List   ${updated_index_list} @{stop_index_list}
Documentation:

Removes all occurrences of given values from list.
Start / End / Elapsed: 20250302 22:58:26.569 / 20250302 22:58:26.569 / 00:00:00.000
00:00:00.000 IF not ${confirm}  
Start / End / Elapsed: 20250302 22:58:26.570 / 20250302 22:58:26.570 / 00:00:00.000
00:06:10.755 FOR ${index} IN @{stop_index_list}  
Start / End / Elapsed: 20250302 22:58:26.570 / 20250302 23:04:37.325 / 00:06:10.755
00:00:08.102 ITERATION ${index} = 1  
Start / End / Elapsed: 20250302 22:58:26.570 / 20250302 22:58:34.672 / 00:00:08.102
00:00:00.487 ITERATION ${index} = 2  
Start / End / Elapsed: 20250302 22:58:34.672 / 20250302 22:58:35.159 / 00:00:00.487
00:06:02.166 ITERATION ${index} = 3  
Start / End / Elapsed: 20250302 22:58:35.159 / 20250302 23:04:37.325 / 00:06:02.166
00:06:02.166 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${timeout} 2s Verify_Karaf_Is_Not_Running_On_Member member_index=${index}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250302 22:58:35.159 / 20250302 23:04:37.325 / 00:06:02.166
Message: 146 failing items removed using the --remove-keywords option.
00:00:00.497 KEYWORD ClusterManagement . Verify_Karaf_Is_Not_Running_On_Member   member_index=${index}
Documentation:

Fail if non-zero karaf instances are counted on member of given index.
Start / End / Elapsed: 20250302 23:04:36.828 / 20250302 23:04:37.325 / 00:00:00.497
00:00:00.495 KEYWORD ${count} = ClusterManagement . Count_Running_Karafs_On_Member   member_index=${member_index}
Documentation:

Remotely execute grep for karaf process, return count as string.
Start / End / Elapsed: 20250302 23:04:36.829 / 20250302 23:04:37.324 / 00:00:00.495
00:00:00.000 KEYWORD BuiltIn . Should Be Equal   0 ${count} Found running Karaf count: ${count}
Documentation:

Fails if the given objects are unequal.
Start / End / Elapsed: 20250302 23:04:37.324 / 20250302 23:04:37.324 / 00:00:00.000
23:04:37.324 FAIL Found running Karaf count: 1: 0 != 1
23:04:37.325 FAIL Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1
00:00:00.000 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=netstat -pnatu | grep 2550
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250302 23:04:37.325 / 20250302 23:04:37.325 / 00:00:00.000
00:00:00.000 RETURN   ${updated_index_list}
Start / End / Elapsed: 20250302 23:04:37.325 / 20250302 23:04:37.325 / 00:00:00.000
00:00:10.652 TEST Unset_Tell_Based_Protocol_Usage
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Unset_Tell_Based_Protocol_Usage
Documentation:

Comment out the flag usage in config file. Also clean most data except data/log/.
Tags: critical
Start / End / Elapsed: 20250302 23:04:37.326 / 20250302 23:04:47.978 / 00:00:10.652
Status: PASS
00:07:22.291 TEST Start_All_And_Sync
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Start_All_And_Sync
Documentation:

Start each member and wait for sync.
Tags: critical
Start / End / Elapsed: 20250302 23:04:47.978 / 20250302 23:12:10.269 / 00:07:22.291
Status: FAIL
Message: Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: ReadTimeout: HTTPConnectionPool(host='10.30.171.139', port=8181): Read timed out. (read timeout=125.0)
00:07:22.285 KEYWORD ClusterManagement . Start_Members_From_List_Or_All  
Documentation:

If the list is empty, start all cluster members. Otherwise, start members based on present indices. If ${wait_for_sync}, wait for cluster sync on listed members. Optionally karaf_home can be overriden. Optionally specific JAVA_HOME is used for starting. Garbage collection is unconditionally logged to files. TODO: Make that reasonable conditional?
Start / End / Elapsed: 20250302 23:04:47.982 / 20250302 23:12:10.267 / 00:07:22.285
00:00:00.000 KEYWORD ${base_command} = BuiltIn . Set Variable If   """${karaf_home}""" != "" ${karaf_home}/bin/start ${NODE_START_COMMAND}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250302 23:04:47.990 / 20250302 23:04:47.990 / 00:00:00.000
00:00:00.000 KEYWORD ${command} = BuiltIn . Set Variable If   """${export_java_home}""" != "" export JAVA_HOME="${export_java_home}"; ${base_command} ${base_command}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250302 23:04:47.990 / 20250302 23:04:47.990 / 00:00:00.000
00:00:00.000 KEYWORD ${epoch} = DateTime . Get Current Date   time_zone=UTC result_format=epoch exclude_millis=False
Documentation:

Returns current local or UTC time with an optional increment.
Start / End / Elapsed: 20250302 23:04:47.991 / 20250302 23:04:47.991 / 00:00:00.000
00:00:00.000 KEYWORD ${gc_filepath} = BuiltIn . Set Variable If   """${karaf_home}""" != "" ${karaf_home}/data/log/gc_${epoch}.log ${GC_LOG_PATH}/gc_${epoch}.log
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250302 23:04:47.991 / 20250302 23:04:47.991 / 00:00:00.000
00:00:00.000 KEYWORD ${gc_options} = BuiltIn . Set Variable If   "docker" not in """${node_start_command}""" -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:${gc_filepath} ${EMPTY}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250302 23:04:47.992 / 20250302 23:04:47.992 / 00:00:00.000
00:00:01.305 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=${command} ${gc_options} member_index_list=${member_index_list}
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250302 23:04:47.992 / 20250302 23:04:49.297 / 00:00:01.305
00:07:18.935 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${timeout} 10s Verify_Members_Are_Ready ${member_index_list} ${wait_for_sync} ${verify_restconf} ${check_system_status} ${service_list}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250302 23:04:49.297 / 20250302 23:12:08.232 / 00:07:18.935
Message: 6 failing items removed using the --remove-keywords option.
00:02:05.273 KEYWORD ClusterManagement . Verify_Members_Are_Ready   ${member_index_list} ${wait_for_sync} ${verify_restconf} ${check_system_status} ${service_list}
Documentation:

Verifies the specified readiness conditions for the given listed members after startup. If ${verify_cluster_sync}, verifies the datastores have synced with the rest of the cluster. If True, verifies RESTCONF is available. If ${verify_system_status}, verifies the system services are OPERATIONAL.
Start / End / Elapsed: 20250302 23:10:02.959 / 20250302 23:12:08.232 / 00:02:05.273
00:02:05.271 IF ${verify_cluster_sync}  
Start / End / Elapsed: 20250302 23:10:02.959 / 20250302 23:12:08.230 / 00:02:05.271
00:02:05.271 KEYWORD ClusterManagement . Check_Cluster_Is_In_Sync   ${member_index_list}
Documentation:

Fail if no-sync is detected on a member from list (or any).
Start / End / Elapsed: 20250302 23:10:02.960 / 20250302 23:12:08.231 / 00:02:05.271
00:00:00.001 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All   given_list=${member_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250302 23:10:02.960 / 20250302 23:10:02.961 / 00:00:00.001
00:02:05.269 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250302 23:10:02.961 / 20250302 23:12:08.230 / 00:02:05.269
00:00:00.082 ITERATION ${index} = 1  
Start / End / Elapsed: 20250302 23:10:02.962 / 20250302 23:10:03.044 / 00:00:00.082
00:00:00.075 ITERATION ${index} = 2  
Start / End / Elapsed: 20250302 23:10:03.043 / 20250302 23:10:03.118 / 00:00:00.075
00:02:05.112 ITERATION ${index} = 3  
Start / End / Elapsed: 20250302 23:10:03.118 / 20250302 23:12:08.230 / 00:02:05.112
00:02:05.112 KEYWORD ${status} = ClusterManagement . Get_Sync_Status_Of_Member   member_index=${index}
Documentation:

Obtain IP, two GETs from jolokia URIs, return combined sync status as string.
Start / End / Elapsed: 20250302 23:10:03.118 / 20250302 23:12:08.230 / 00:02:05.112
00:00:00.001 KEYWORD ${session} = ClusterManagement . Resolve_Http_Session_For_Member   member_index=${member_index}
Documentation:

Return RequestsLibrary session alias pointing to node of given index.
Start / End / Elapsed: 20250302 23:10:03.119 / 20250302 23:10:03.120 / 00:00:00.001
00:02:05.110 KEYWORD ${conf_text} = TemplatedRequests . Get_As_Json_From_Uri   uri=${JOLOKIA_CONF_SHARD_MANAGER_URI} session=${session}
Documentation:

Specify JSON headers and return Get_From_Uri normalized response text.
Start / End / Elapsed: 20250302 23:10:03.120 / 20250302 23:12:08.230 / 00:02:05.110
00:02:05.107 KEYWORD ${response_text} = TemplatedRequests . Get_From_Uri   uri=${uri} accept=${ACCEPT_EMPTY} session=${session} normalize_json=True http_timeout=${http_timeout} log_response=${log_response}
Documentation:

GET data from given URI, check status code and return response text. ${accept} is a Python object with headers to use. If ${normalize_json}, normalize as JSON text before returning.
Start / End / Elapsed: 20250302 23:10:03.122 / 20250302 23:12:08.229 / 00:02:05.107
00:00:00.000 KEYWORD BuiltIn . Log   ${uri}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250302 23:10:03.126 / 20250302 23:10:03.126 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${accept}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250302 23:10:03.126 / 20250302 23:10:03.126 / 00:00:00.000
00:02:05.101 IF """${http_timeout}""" == """${EMPTY}"""  
Start / End / Elapsed: 20250302 23:10:03.126 / 20250302 23:12:08.227 / 00:02:05.101
00:00:00.000 ELSE  
Start / End / Elapsed: 20250302 23:12:08.227 / 20250302 23:12:08.227 / 00:00:00.000
00:00:00.000 KEYWORD TemplatedRequests . Check_Status_Code   ${response} log_response=${log_response}
Documentation:

Log response text, check status_code is one of allowed ones. In cases where this keyword is called in a WUKS it could end up logging tons of data and it may be desired to skip the logging by passing log_response=False, but by default it remains True.
Start / End / Elapsed: 20250302 23:12:08.228 / 20250302 23:12:08.228 / 00:00:00.000
00:00:00.000 IF not ${normalize_json}  
Start / End / Elapsed: 20250302 23:12:08.229 / 20250302 23:12:08.229 / 00:00:00.000
00:00:00.000 KEYWORD ${text_normalized} = norm_json . Normalize Json Text   ${response.text} jmes_path=${jmes_path} keys_with_volatiles=${keys_with_volatiles}
Documentation:

Attempt to return sorted indented JSON string.
Start / End / Elapsed: 20250302 23:12:08.229 / 20250302 23:12:08.229 / 00:00:00.000
00:00:00.000 RETURN   ${text_normalized}
Start / End / Elapsed: 20250302 23:12:08.229 / 20250302 23:12:08.229 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250302 23:12:08.229 / 20250302 23:12:08.229 / 00:00:00.000
00:00:00.000 KEYWORD ${conf_status} = ClusterManagement . ClusterManagement__Parse_Sync_Status   shard_manager_text=${conf_text}
Documentation:

Return sync status parsed out of given text. Called twice by Get_Sync_Status_Of_Member.
Start / End / Elapsed: 20250302 23:12:08.229 / 20250302 23:12:08.229 / 00:00:00.000
00:00:00.000 IF 'False' == ${conf_status}  
Start / End / Elapsed: 20250302 23:12:08.229 / 20250302 23:12:08.229 / 00:00:00.000
00:00:00.000 KEYWORD ${oper_text} = TemplatedRequests . Get_As_Json_From_Uri   uri=${JOLOKIA_OPER_SHARD_MANAGER_URI} session=${session}
Documentation:

Specify JSON headers and return Get_From_Uri normalized response text.
Start / End / Elapsed: 20250302 23:12:08.230 / 20250302 23:12:08.230 / 00:00:00.000
00:00:00.000 KEYWORD ${oper_status} = ClusterManagement . ClusterManagement__Parse_Sync_Status   shard_manager_text=${oper_text}
Documentation:

Return sync status parsed out of given text. Called twice by Get_Sync_Status_Of_Member.
Start / End / Elapsed: 20250302 23:12:08.230 / 20250302 23:12:08.230 / 00:00:00.000
00:00:00.000 RETURN   ${oper_status}
Start / End / Elapsed: 20250302 23:12:08.230 / 20250302 23:12:08.230 / 00:00:00.000
00:00:00.000 IF 'True' == '${status}'  
Start / End / Elapsed: 20250302 23:12:08.230 / 20250302 23:12:08.230 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Fail   Index ${index} has incorrect status: ${status}
Documentation:

Fails the test with the given message and optionally alters its tags.
Start / End / Elapsed: 20250302 23:12:08.230 / 20250302 23:12:08.230 / 00:00:00.000
00:00:00.000 IF ${verify_restconf}  
Start / End / Elapsed: 20250302 23:12:08.231 / 20250302 23:12:08.231 / 00:00:00.000
00:00:00.000 IF ${verify_system_status} and ("${service_list}" != "[[]]")  
Start / End / Elapsed: 20250302 23:12:08.231 / 20250302 23:12:08.231 / 00:00:00.000
23:12:08.232 FAIL Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: ReadTimeout: HTTPConnectionPool(host='10.30.171.139', port=8181): Read timed out. (read timeout=125.0)
00:00:02.035 TEARDOWN ClusterManagement . Run_Bash_Command_On_List_Or_All   command=netstat -pnatu | grep 2550
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250302 23:12:08.232 / 20250302 23:12:10.267 / 00:00:02.035
00:00:00.000 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 10s ClusterManagement.Run_Bash_Command_On_List_Or_All netstat -punta
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250302 23:12:10.268 / 20250302 23:12:10.268 / 00:00:00.000
00:00:00.000 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All  
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250302 23:12:10.268 / 20250302 23:12:10.268 / 00:00:00.000
00:00:00.001 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250302 23:12:10.268 / 20250302 23:12:10.269 / 00:00:00.001
00:00:00.000 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 10s ShardStability.Shards_Stability_Get_Details ${DEFAULT_SHARD_LIST} verify_restconf=True
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250302 23:12:10.269 / 20250302 23:12:10.269 / 00:00:00.000
00:14:40.702 SUITE Rpc Provider Precedence
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence
Documentation:

DOMRpcBroker testing: RPC Provider Precedence

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

The aim is to establish that remote RPC implementations have lower priority than local ones, which is to say that any movement of RPCs on remote nodes does not affect routing as long as a local implementation is available.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot
Start / End / Elapsed: 20250302 23:12:10.270 / 20250302 23:26:50.972 / 00:14:40.702
Status: 8 tests total, 4 passed, 4 failed, 0 skipped
00:13:24.223 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250302 23:26:50.973 / 20250302 23:40:15.196 / 00:13:24.223
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:09:20.089 SUITE Rpc Provider Partition And Heal
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal
Documentation:

DOMRpcBroker testing: RPC Provider Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This tests establishes that the RPC service operates correctly when faced with node failures. This suite supports more than three node cluster setup too.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot
Start / End / Elapsed: 20250302 23:40:15.197 / 20250302 23:49:35.286 / 00:09:20.089
Status: 7 tests total, 0 passed, 7 failed, 0 skipped
00:13:24.110 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250302 23:49:35.287 / 20250303 00:02:59.397 / 00:13:24.110
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:06:18.533 SUITE Action Provider Precedence
Full Name: controller-clustering-ask.txt.Action Provider Precedence
Documentation:

DOMRpcBroker testing: Action Provider Precedence

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

The aim is to establish that remote action implementations have lower priority than local ones, which is to say that any movement of actions on remote nodes does not affect routing as long as a local implementation is available.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot
Start / End / Elapsed: 20250303 00:02:59.399 / 20250303 00:09:17.932 / 00:06:18.533
Status: 8 tests total, 3 passed, 5 failed, 0 skipped
00:13:24.370 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250303 00:09:17.932 / 20250303 00:22:42.302 / 00:13:24.370
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:09:19.664 SUITE Action Provider Partition And Heal
Full Name: controller-clustering-ask.txt.Action Provider Partition And Heal
Documentation:

DOMRpcBroker testing: RPC Action Provider Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This tests establishes that the RPC service for actions operates correctly when faced with node failures. This suite supports more than three node cluster setup too.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot
Start / End / Elapsed: 20250303 00:22:42.302 / 20250303 00:32:01.966 / 00:09:19.664
Status: 7 tests total, 0 passed, 7 failed, 0 skipped
00:13:25.239 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250303 00:32:01.967 / 20250303 00:45:27.206 / 00:13:25.239
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:04:56.686 SUITE Master Stability
Full Name: controller-clustering-ask.txt.Master Stability
Documentation:

Cluster Singleton testing: Master Stability

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Cluster Singleton service is designed to ensure that only one instance of an application is registered globally in the cluster. The goal is to establish the service operates correctly in face of application registration changing without moving the active instance.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/cluster_singleton/master_stability.robot
Start / End / Elapsed: 20250303 00:45:27.207 / 20250303 00:50:23.893 / 00:04:56.686
Status: 6 tests total, 0 passed, 6 failed, 0 skipped
00:13:24.835 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250303 00:50:23.894 / 20250303 01:03:48.729 / 00:13:24.835
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:04:57.119 SUITE Partition And Heal
Full Name: controller-clustering-ask.txt.Partition And Heal
Documentation:

Cluster Singleton testing: Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Cluster Singleton service is designed to ensure that only one instance of an application is registered globally in the cluster. The goal is to establish the service operates correctly in face of node failures.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot
Start / End / Elapsed: 20250303 01:03:48.731 / 20250303 01:08:45.850 / 00:04:57.119
Status: 6 tests total, 0 passed, 6 failed, 0 skipped
00:13:23.739 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250303 01:08:45.850 / 20250303 01:22:09.589 / 00:13:23.739
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:03:55.966 SUITE Chasing The Leader
Full Name: controller-clustering-ask.txt.Chasing The Leader
Documentation:

Cluster Singleton testing: Chasing the Leader

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test aims to establish the service operates correctly when faced with rapid application transitions without having a stabilized application.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot
Start / End / Elapsed: 20250303 01:22:09.590 / 20250303 01:26:05.556 / 00:03:55.966
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:13:21.201 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250303 01:26:05.556 / 20250303 01:39:26.757 / 00:13:21.201
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:01:04.581 SUITE Global Rpc Kill
Full Name: controller-clustering-ask.txt.Global Rpc Kill
Documentation:

Controller functional HA testing of global singleton rpcs.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot
Start / End / Elapsed: 20250303 01:39:26.758 / 20250303 01:40:31.339 / 00:01:04.581
Status: 11 tests total, 0 passed, 11 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:23.845 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250303 01:40:31.340 / 20250303 01:53:55.185 / 00:13:23.845
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:01:04.428 SUITE Global Rpc Freeze
Full Name: controller-clustering-ask.txt.Global Rpc Freeze
Documentation:

Controller functional HA testing of global singleton rpcs if jvm frozen.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot
Start / End / Elapsed: 20250303 01:53:55.185 / 20250303 01:54:59.613 / 00:01:04.428
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:21.849 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250303 01:54:59.613 / 20250303 02:08:21.462 / 00:13:21.849
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:01:04.458 SUITE Global Rpc Isolate
Full Name: controller-clustering-ask.txt.Global Rpc Isolate
Documentation:

Controller functional HA testing of global singleton rpcs.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot
Start / End / Elapsed: 20250303 02:08:21.463 / 20250303 02:09:25.921 / 00:01:04.458
Status: 11 tests total, 0 passed, 11 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:25.868 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250303 02:09:25.922 / 20250303 02:22:51.790 / 00:13:25.868
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:54.309 SUITE Carpeople Crud
Full Name: controller-clustering-ask.txt.Carpeople Crud
Documentation:

Suite for performing basic car/people CRUD operations on leaders and followers.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

More precisely, Update operation is not executed, but some operations are using specific RPCs which goes beyond "basic CRUD".

Cars are added by one big PUT to datastore on car Leader. People are added in a loop with add-person RPC on a people Follower. Cars are bought by chunks on each member, by loop with buy-car RPC.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot
Start / End / Elapsed: 20250303 02:22:51.791 / 20250303 02:23:46.100 / 00:00:54.309
Status: 13 tests total, 0 passed, 13 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.115:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:52.382 SUITE Car Failover Crud
Full Name: controller-clustering-ask.txt.Car Failover Crud
Documentation:

Suite mixing basic operations with restart of car Leader.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test stops the current leader of the "car" shard and then executes CRD operations on the new leader and a new follower. The stopped member is brought back. This suite uses 3 different car sets, same size but different starting ID.

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot
Start / End / Elapsed: 20250303 02:23:46.100 / 20250303 02:24:38.482 / 00:00:52.382
Status: 16 tests total, 0 passed, 16 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.115:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:53.136 SUITE Car Persistence Recovery
Full Name: controller-clustering-ask.txt.Car Persistence Recovery
Documentation:

This test restarts all controllers to verify recovery of car data from persistence.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot
Start / End / Elapsed: 20250303 02:24:38.483 / 20250303 02:25:31.619 / 00:00:53.136
Status: 7 tests total, 0 passed, 7 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.115:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:52.308 SUITE Buycar Failover
Full Name: controller-clustering-ask.txt.Buycar Failover
Documentation:

This test focuses on testing buy-car RPC over 3 Leader reboots.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

All purchases are against the same node, which is the first one to get rebooted.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot
Start / End / Elapsed: 20250303 02:25:31.619 / 20250303 02:26:23.927 / 00:00:52.308
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.115:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:54.293 SUITE Entity Isolate
Full Name: controller-clustering-ask.txt.Entity Isolate
Documentation:

Suite for performing member isolation and rejoin, we do with entity-ownership leader.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot
Start / End / Elapsed: 20250303 02:26:23.927 / 20250303 02:27:18.220 / 00:00:54.293
Status: 5 tests total, 0 passed, 5 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.115:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:52.286 SUITE Buycar Failover Isolation
Full Name: controller-clustering-ask.txt.Buycar Failover Isolation
Documentation:

This test focuses on testing buy-car RPC over the Leader isolation.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

All purchases are against the same node, which is the first one to get isolated.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot
Start / End / Elapsed: 20250303 02:27:18.222 / 20250303 02:28:10.508 / 00:00:52.286
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.115:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:52.986 SUITE Car Failover Crud Isolation
Full Name: controller-clustering-ask.txt.Car Failover Crud Isolation
Documentation:

Suite mixing basic operations with isolation of car Leader.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test isolates the current leader of the "car" shard and then executes CRD operations on the new leader and a new follower. The isolated member is brought back. This suite uses 3 different car sets, same size but different starting ID.

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot
Start / End / Elapsed: 20250303 02:28:10.508 / 20250303 02:29:03.494 / 00:00:52.986
Status: 16 tests total, 0 passed, 16 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.115:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:52.327 SUITE Car Outage Corners
Full Name: controller-clustering-ask.txt.Car Outage Corners
Documentation:

Cluster suite for testing minimal and sum-minimal member population behavior.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test stops majority of the followers and verifies car addition is not possible, then resumes single follower (first from original list) and checks that addition works. Then remaining members are brought up. Leader member is always up and assumed to remain Leading during the whole suite run.

TODO: Use initial data to check more operations. TODO: Perhaps merge with car_failover_crud suite.

Other modules and Shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot
Start / End / Elapsed: 20250303 02:29:03.494 / 20250303 02:29:55.821 / 00:00:52.327
Status: 8 tests total, 0 passed, 8 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.115:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig