Test Statistics

Total Statistics
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
All Tests
19529166002:28:10
Statistics by Tag
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
carpeople
85085000:00:00
clustering
85085000:00:00
critical
19529166002:28:10
Statistics by Suite
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
controller-clustering-ask.txt
19529166002:43:23
controller-clustering-ask . txt . Restart Odl With Tell Based False
321000:05:28
controller-clustering-ask . txt . Rpc Provider Precedence
880000:00:11
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:12:38
controller-clustering-ask . txt . Rpc Provider Partition And Heal
725000:04:33
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:09
controller-clustering-ask . txt . Action Provider Precedence
844000:03:20
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:10
controller-clustering-ask . txt . Action Provider Partition And Heal
725000:04:34
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:06
controller-clustering-ask . txt . Master Stability
606000:02:38
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:09
controller-clustering-ask . txt . Partition And Heal
606000:02:38
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:08
controller-clustering-ask . txt . Chasing The Leader
312000:02:37
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:08
controller-clustering-ask . txt . Global Rpc Kill
11011000:00:45
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:07
controller-clustering-ask . txt . Global Rpc Freeze
10010000:00:44
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:05
controller-clustering-ask . txt . Global Rpc Isolate
11011000:00:44
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:03
controller-clustering-ask . txt . Carpeople Crud
13013000:00:35
controller-clustering-ask . txt . Car Failover Crud
16016000:00:33
controller-clustering-ask . txt . Car Persistence Recovery
707000:00:33
controller-clustering-ask . txt . Buycar Failover
10010000:00:33
controller-clustering-ask . txt . Entity Isolate
505000:00:35
controller-clustering-ask . txt . Buycar Failover Isolation
10010000:00:33
controller-clustering-ask . txt . Car Failover Crud Isolation
16016000:00:33
controller-clustering-ask . txt . Car Outage Corners
808000:00:33

Test Execution Log

02:43:22.558 SUITE controller-clustering-ask.txt
Full Name: controller-clustering-ask.txt
Start / End / Elapsed: 20250222 23:38:27.407 / 20250223 02:21:49.965 / 02:43:22.558
Status: 195 tests total, 29 passed, 166 failed, 0 skipped
00:05:27.991 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250222 23:38:27.430 / 20250222 23:43:55.421 / 00:05:27.991
Status: 3 tests total, 2 passed, 1 failed, 0 skipped
00:00:04.814 SETUP SetupUtils . Setup_Utils_For_Setup_And_Teardown   http_timeout=125
Documentation:

Prepare both FailFast and Karaf logging, log that the caller suite has started. This keyword is to be used in suite setup, and without it the other keywords may not work properly.
Start / End / Elapsed: 20250222 23:38:27.975 / 20250222 23:38:32.789 / 00:00:04.814
00:00:00.000 TEARDOWN SSHLibrary . Close All Connections  
Documentation:

Closes all open connections.
Start / End / Elapsed: 20250222 23:43:55.421 / 20250222 23:43:55.421 / 00:00:00.000
00:04:11.201 TEST Stop_All_Members
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Stop_All_Members
Documentation:

Stop every odl node.
Tags: critical
Start / End / Elapsed: 20250222 23:38:32.789 / 20250222 23:42:43.990 / 00:04:11.201
Status: PASS
00:00:09.954 TEST Unset_Tell_Based_Protocol_Usage
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Unset_Tell_Based_Protocol_Usage
Documentation:

Comment out the flag usage in config file. Also clean most data except data/log/.
Tags: critical
Start / End / Elapsed: 20250222 23:42:43.990 / 20250222 23:42:53.944 / 00:00:09.954
Status: PASS
00:01:01.475 TEST Start_All_And_Sync
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Start_All_And_Sync
Documentation:

Start each member and wait for sync.
Tags: critical
Start / End / Elapsed: 20250222 23:42:53.945 / 20250222 23:43:55.420 / 00:01:01.475
Status: FAIL
Message: Got rc: 1 or stderr was not empty: sudo: netstat: command not found
00:00:59.641 KEYWORD ClusterManagement . Start_Members_From_List_Or_All  
Documentation:

If the list is empty, start all cluster members. Otherwise, start members based on present indices. If ${wait_for_sync}, wait for cluster sync on listed members. Optionally karaf_home can be overriden. Optionally specific JAVA_HOME is used for starting. Garbage collection is unconditionally logged to files. TODO: Make that reasonable conditional?
Start / End / Elapsed: 20250222 23:42:53.950 / 20250222 23:43:53.591 / 00:00:59.641
00:00:01.380 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 10s ClusterManagement.Run_Bash_Command_On_List_Or_All netstat -punta
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250222 23:43:53.591 / 20250222 23:43:54.971 / 00:00:01.380
00:00:00.003 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All  
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250222 23:43:54.971 / 20250222 23:43:54.974 / 00:00:00.003
00:00:00.446 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250222 23:43:54.974 / 20250222 23:43:55.420 / 00:00:00.446
00:00:00.446 ITERATION ${index} = 1  
Start / End / Elapsed: 20250222 23:43:54.974 / 20250222 23:43:55.420 / 00:00:00.446
00:00:00.444 KEYWORD ${output} = ClusterManagement . Check_Bash_Command_On_Member   command=sudo netstat -punta | grep 2550 | grep LISTEN member_index=${index}
Documentation:

Open SSH session, call SSHKeywords.Execute_Command_Passes, close session, restore previously active session and return output.
Start / End / Elapsed: 20250222 23:43:54.974 / 20250222 23:43:55.418 / 00:00:00.444
00:00:00.444 KEYWORD BuiltIn . Run Keyword And Return   SSHKeywords.Run_Keyword_Preserve_Connection Check_Unsafely_Bash_Command_On_Member ${command} ${member_index} return_success_only=${return_success_only} log_on_success=${log_on_success} log_on_failure=${log_on_failure} stderr_must_be_empty=${stderr_must_be_empty}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250222 23:43:54.975 / 20250222 23:43:55.419 / 00:00:00.444
00:00:00.443 KEYWORD SSHKeywords . Run_Keyword_Preserve_Connection   Check_Unsafely_Bash_Command_On_Member ${command} ${member_index} return_success_only=${return_success_only} log_on_success=${log_on_success} log_on_failure=${log_on_failure} stderr_must_be_empty=${stderr_must_be_empty}
Documentation:

Store current connection index, run keyword returning its result, restore connection in teardown. Note that in order to avoid "got positional argument after named arguments", it is safer to use positional (not named) arguments on call.
Start / End / Elapsed: 20250222 23:43:54.975 / 20250222 23:43:55.418 / 00:00:00.443
00:00:00.000 KEYWORD ${current_connection} = SSHLibrary . Get Connection  
Documentation:

Returns information about the connection.
Start / End / Elapsed: 20250222 23:43:54.976 / 20250222 23:43:54.976 / 00:00:00.000
00:00:00.437 KEYWORD BuiltIn . Run Keyword And Return   ${keyword_name} @{args} &{kwargs}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250222 23:43:54.976 / 20250222 23:43:55.413 / 00:00:00.437
00:00:00.437 KEYWORD ClusterManagement . Check_Unsafely_Bash_Command_On_Member   @{args} &{kwargs}
Documentation:

Obtain Ip address, open session, call SSHKeywords.Execute_Command_Passes, close session and return output. This affects which SSH session is active.
Start / End / Elapsed: 20250222 23:43:54.976 / 20250222 23:43:55.413 / 00:00:00.437
00:00:00.001 KEYWORD ${member_ip} = ClusterManagement . Resolve_IP_Address_For_Member   ${member_index}
Documentation:

Return node IP address of given index.
Start / End / Elapsed: 20250222 23:43:54.977 / 20250222 23:43:54.978 / 00:00:00.001
00:00:00.435 KEYWORD BuiltIn . Run Keyword And Return   SSHKeywords.Run_Unsafely_Keyword_Over_Temporary_Odl_Session ${member_ip} Execute_Command_Passes ${command} return_success_only=${return_success_only} log_on_success=${log_on_success} log_on_failure=${log_on_failure} stderr_must_be_empty=${stderr_must_be_empty}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250222 23:43:54.978 / 20250222 23:43:55.413 / 00:00:00.435
00:00:00.435 KEYWORD SSHKeywords . Run_Unsafely_Keyword_Over_Temporary_Odl_Session   ${member_ip} Execute_Command_Passes ${command} return_success_only=${return_success_only} log_on_success=${log_on_success} log_on_failure=${log_on_failure} stderr_must_be_empty=${stderr_must_be_empty}
Documentation:

Open connection to given IP address, run keyword, close connection, return result. This is unsafe in the sense that previously active session will be switched out off, but safe in the sense only the temporary connection is closed.
Start / End / Elapsed: 20250222 23:43:54.978 / 20250222 23:43:55.413 / 00:00:00.435
00:00:00.407 KEYWORD SSHKeywords . Open_Connection_To_ODL_System   ${ip_address}
Documentation:

Open a connection to the ODL system at 10.30.170.51 and return its identifier.
Start / End / Elapsed: 20250222 23:43:54.979 / 20250222 23:43:55.386 / 00:00:00.407
00:00:00.025 KEYWORD ${status} ${result} = BuiltIn . Run Keyword And Ignore Error   ${keyword_name} @{args} &{kwargs}
Documentation:

Runs the given keyword with the given arguments and ignores possible error.
Start / End / Elapsed: 20250222 23:43:55.386 / 20250222 23:43:55.411 / 00:00:00.025
00:00:00.000 KEYWORD SSHLibrary . Close Connection  
Documentation:

Closes the current connection.
Start / End / Elapsed: 20250222 23:43:55.411 / 20250222 23:43:55.411 / 00:00:00.000
00:00:00.000 IF "${status}" == "PASS"  
Start / End / Elapsed: 20250222 23:43:55.412 / 20250222 23:43:55.412 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Fail   ${result}
Documentation:

Fails the test with the given message and optionally alters its tags.
Start / End / Elapsed: 20250222 23:43:55.412 / 20250222 23:43:55.412 / 00:00:00.000
23:43:55.412 FAIL Got rc: 1 or stderr was not empty: sudo: netstat: command not found
23:43:55.413 INFO Returning from the enclosing user keyword.
23:43:55.413 INFO Returning from the enclosing user keyword.
00:00:00.005 TEARDOWN SSHKeywords . Restore_Current_Ssh_Connection_From_Index   ${current_connection.index}
Documentation:

Restore active SSH connection in SSHLibrary to given index.
Start / End / Elapsed: 20250222 23:43:55.414 / 20250222 23:43:55.419 / 00:00:00.005
23:43:55.419 INFO Returning from the enclosing user keyword.
00:00:00.000 KEYWORD ${listening} = Get Match   ${output} LISTEN
Documentation:

Wrapper around String.Get Regexp Matches to return None if not found or the first match if found.
Start / End / Elapsed: 20250222 23:43:55.419 / 20250222 23:43:55.419 / 00:00:00.000
00:00:00.001 IF '${listening}' == 'None'  
Start / End / Elapsed: 20250222 23:43:55.419 / 20250222 23:43:55.420 / 00:00:00.001
00:00:00.000 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 10s ShardStability.Shards_Stability_Get_Details ${DEFAULT_SHARD_LIST} verify_restconf=True
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250222 23:43:55.420 / 20250222 23:43:55.420 / 00:00:00.000
00:00:10.803 SUITE Rpc Provider Precedence
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence
Documentation:

DOMRpcBroker testing: RPC Provider Precedence

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

The aim is to establish that remote RPC implementations have lower priority than local ones, which is to say that any movement of RPCs on remote nodes does not affect routing as long as a local implementation is available.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot
Start / End / Elapsed: 20250222 23:43:55.422 / 20250222 23:44:06.225 / 00:00:10.803
Status: 8 tests total, 8 passed, 0 failed, 0 skipped
00:12:38.489 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250222 23:44:06.226 / 20250222 23:56:44.715 / 00:12:38.489
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:02.461 SETUP SetupUtils . Setup_Utils_For_Setup_And_Teardown   http_timeout=125
Documentation:

Prepare both FailFast and Karaf logging, log that the caller suite has started. This keyword is to be used in suite setup, and without it the other keywords may not work properly.
Start / End / Elapsed: 20250222 23:44:06.321 / 20250222 23:44:08.782 / 00:00:02.461
00:00:00.000 TEARDOWN SSHLibrary . Close All Connections  
Documentation:

Closes all open connections.
Start / End / Elapsed: 20250222 23:56:44.715 / 20250222 23:56:44.715 / 00:00:00.000
00:06:12.424 TEST Stop_All_Members
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Stop_All_Members
Documentation:

Stop every odl node.
Tags: critical
Start / End / Elapsed: 20250222 23:44:08.782 / 20250222 23:50:21.206 / 00:06:12.424
Status: FAIL
Message: Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1
00:06:12.423 KEYWORD ClusterManagement . Stop_Members_From_List_Or_All  
Documentation:

If the list is empty, stops all ODL instances. Otherwise stop members based on ${stop_index_list} If ${confirm} is True, verify stopped instances are not there anymore. The KW will return a list of available members: ${updated index_list}=${original_index_list}-${member_index_list}
Start / End / Elapsed: 20250222 23:44:08.783 / 20250222 23:50:21.206 / 00:06:12.423
00:00:00.001 KEYWORD ${stop_index_list} = ClusterManagement . List_Indices_Or_All   given_list=${member_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250222 23:44:08.788 / 20250222 23:44:08.789 / 00:00:00.001
00:00:00.001 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All   given_list=${original_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250222 23:44:08.789 / 20250222 23:44:08.790 / 00:00:00.001
00:00:02.547 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=${NODE_STOP_COMMAND} member_index_list=${member_index_list}
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250222 23:44:08.790 / 20250222 23:44:11.337 / 00:00:02.547
00:00:00.000 KEYWORD ${updated_index_list} = BuiltIn . Create List   @{index_list}
Documentation:

Returns a list containing given items.
Start / End / Elapsed: 20250222 23:44:11.338 / 20250222 23:44:11.338 / 00:00:00.000
00:00:00.000 KEYWORD Collections . Remove Values From List   ${updated_index_list} @{stop_index_list}
Documentation:

Removes all occurrences of given values from list.
Start / End / Elapsed: 20250222 23:44:11.338 / 20250222 23:44:11.338 / 00:00:00.000
00:00:00.000 IF not ${confirm}  
Start / End / Elapsed: 20250222 23:44:11.338 / 20250222 23:44:11.338 / 00:00:00.000
00:06:09.867 FOR ${index} IN @{stop_index_list}  
Start / End / Elapsed: 20250222 23:44:11.339 / 20250222 23:50:21.206 / 00:06:09.867
00:00:07.960 ITERATION ${index} = 1  
Start / End / Elapsed: 20250222 23:44:11.339 / 20250222 23:44:19.299 / 00:00:07.960
00:00:00.439 ITERATION ${index} = 2  
Start / End / Elapsed: 20250222 23:44:19.298 / 20250222 23:44:19.737 / 00:00:00.439
00:06:01.468 ITERATION ${index} = 3  
Start / End / Elapsed: 20250222 23:44:19.737 / 20250222 23:50:21.205 / 00:06:01.468
00:06:01.468 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${timeout} 2s Verify_Karaf_Is_Not_Running_On_Member member_index=${index}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250222 23:44:19.737 / 20250222 23:50:21.205 / 00:06:01.468
Message: 146 failing items removed using the --remove-keywords option.
00:00:00.464 KEYWORD ClusterManagement . Verify_Karaf_Is_Not_Running_On_Member   member_index=${index}
Documentation:

Fail if non-zero karaf instances are counted on member of given index.
Start / End / Elapsed: 20250222 23:50:20.742 / 20250222 23:50:21.206 / 00:00:00.464
00:00:00.462 KEYWORD ${count} = ClusterManagement . Count_Running_Karafs_On_Member   member_index=${member_index}
Documentation:

Remotely execute grep for karaf process, return count as string.
Start / End / Elapsed: 20250222 23:50:20.743 / 20250222 23:50:21.205 / 00:00:00.462
00:00:00.000 KEYWORD BuiltIn . Should Be Equal   0 ${count} Found running Karaf count: ${count}
Documentation:

Fails if the given objects are unequal.
Start / End / Elapsed: 20250222 23:50:21.205 / 20250222 23:50:21.205 / 00:00:00.000
23:50:21.205 FAIL Found running Karaf count: 1: 0 != 1
23:50:21.205 FAIL Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1
00:00:00.000 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=netstat -pnatu | grep 2550
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250222 23:50:21.206 / 20250222 23:50:21.206 / 00:00:00.000
00:00:00.000 RETURN   ${updated_index_list}
Start / End / Elapsed: 20250222 23:50:21.206 / 20250222 23:50:21.206 / 00:00:00.000
00:00:10.643 TEST Unset_Tell_Based_Protocol_Usage
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Unset_Tell_Based_Protocol_Usage
Documentation:

Comment out the flag usage in config file. Also clean most data except data/log/.
Tags: critical
Start / End / Elapsed: 20250222 23:50:21.206 / 20250222 23:50:31.849 / 00:00:10.643
Status: PASS
00:06:12.865 TEST Start_All_And_Sync
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Start_All_And_Sync
Documentation:

Start each member and wait for sync.
Tags: critical
Start / End / Elapsed: 20250222 23:50:31.849 / 20250222 23:56:44.714 / 00:06:12.865
Status: FAIL
Message: Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore
00:06:12.858 KEYWORD ClusterManagement . Start_Members_From_List_Or_All  
Documentation:

If the list is empty, start all cluster members. Otherwise, start members based on present indices. If ${wait_for_sync}, wait for cluster sync on listed members. Optionally karaf_home can be overriden. Optionally specific JAVA_HOME is used for starting. Garbage collection is unconditionally logged to files. TODO: Make that reasonable conditional?
Start / End / Elapsed: 20250222 23:50:31.854 / 20250222 23:56:44.712 / 00:06:12.858
00:00:00.000 KEYWORD ${base_command} = BuiltIn . Set Variable If   """${karaf_home}""" != "" ${karaf_home}/bin/start ${NODE_START_COMMAND}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250222 23:50:31.862 / 20250222 23:50:31.862 / 00:00:00.000
00:00:00.000 KEYWORD ${command} = BuiltIn . Set Variable If   """${export_java_home}""" != "" export JAVA_HOME="${export_java_home}"; ${base_command} ${base_command}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250222 23:50:31.863 / 20250222 23:50:31.863 / 00:00:00.000
00:00:00.000 KEYWORD ${epoch} = DateTime . Get Current Date   time_zone=UTC result_format=epoch exclude_millis=False
Documentation:

Returns current local or UTC time with an optional increment.
Start / End / Elapsed: 20250222 23:50:31.863 / 20250222 23:50:31.863 / 00:00:00.000
00:00:00.000 KEYWORD ${gc_filepath} = BuiltIn . Set Variable If   """${karaf_home}""" != "" ${karaf_home}/data/log/gc_${epoch}.log ${GC_LOG_PATH}/gc_${epoch}.log
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250222 23:50:31.863 / 20250222 23:50:31.863 / 00:00:00.000
00:00:00.000 KEYWORD ${gc_options} = BuiltIn . Set Variable If   "docker" not in """${node_start_command}""" -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:${gc_filepath} ${EMPTY}
Documentation:

Sets variable based on the given condition.
Start / End / Elapsed: 20250222 23:50:31.864 / 20250222 23:50:31.864 / 00:00:00.000
00:00:01.373 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=${command} ${gc_options} member_index_list=${member_index_list}
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250222 23:50:31.864 / 20250222 23:50:33.237 / 00:00:01.373
00:06:09.452 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${timeout} 10s Verify_Members_Are_Ready ${member_index_list} ${wait_for_sync} ${verify_restconf} ${check_system_status} ${service_list}
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250222 23:50:33.238 / 20250222 23:56:42.690 / 00:06:09.452
Message: 36 failing items removed using the --remove-keywords option.
00:00:00.183 KEYWORD ClusterManagement . Verify_Members_Are_Ready   ${member_index_list} ${wait_for_sync} ${verify_restconf} ${check_system_status} ${service_list}
Documentation:

Verifies the specified readiness conditions for the given listed members after startup. If ${verify_cluster_sync}, verifies the datastores have synced with the rest of the cluster. If True, verifies RESTCONF is available. If ${verify_system_status}, verifies the system services are OPERATIONAL.
Start / End / Elapsed: 20250222 23:56:42.506 / 20250222 23:56:42.689 / 00:00:00.183
00:00:00.181 IF ${verify_cluster_sync}  
Start / End / Elapsed: 20250222 23:56:42.507 / 20250222 23:56:42.688 / 00:00:00.181
00:00:00.181 KEYWORD ClusterManagement . Check_Cluster_Is_In_Sync   ${member_index_list}
Documentation:

Fail if no-sync is detected on a member from list (or any).
Start / End / Elapsed: 20250222 23:56:42.507 / 20250222 23:56:42.688 / 00:00:00.181
00:00:00.001 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All   given_list=${member_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250222 23:56:42.507 / 20250222 23:56:42.508 / 00:00:00.001
00:00:00.179 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250222 23:56:42.509 / 20250222 23:56:42.688 / 00:00:00.179
00:00:00.077 ITERATION ${index} = 1  
Start / End / Elapsed: 20250222 23:56:42.509 / 20250222 23:56:42.586 / 00:00:00.077
00:00:00.074 ITERATION ${index} = 2  
Start / End / Elapsed: 20250222 23:56:42.586 / 20250222 23:56:42.660 / 00:00:00.074
00:00:00.027 ITERATION ${index} = 3  
Start / End / Elapsed: 20250222 23:56:42.660 / 20250222 23:56:42.687 / 00:00:00.027
00:00:00.026 KEYWORD ${status} = ClusterManagement . Get_Sync_Status_Of_Member   member_index=${index}
Documentation:

Obtain IP, two GETs from jolokia URIs, return combined sync status as string.
Start / End / Elapsed: 20250222 23:56:42.660 / 20250222 23:56:42.686 / 00:00:00.026
00:00:00.001 KEYWORD ${session} = ClusterManagement . Resolve_Http_Session_For_Member   member_index=${member_index}
Documentation:

Return RequestsLibrary session alias pointing to node of given index.
Start / End / Elapsed: 20250222 23:56:42.661 / 20250222 23:56:42.662 / 00:00:00.001
00:00:00.022 KEYWORD ${conf_text} = TemplatedRequests . Get_As_Json_From_Uri   uri=${JOLOKIA_CONF_SHARD_MANAGER_URI} session=${session}
Documentation:

Specify JSON headers and return Get_From_Uri normalized response text.
Start / End / Elapsed: 20250222 23:56:42.662 / 20250222 23:56:42.684 / 00:00:00.022
00:00:00.020 KEYWORD ${response_text} = TemplatedRequests . Get_From_Uri   uri=${uri} accept=${ACCEPT_EMPTY} session=${session} normalize_json=True http_timeout=${http_timeout} log_response=${log_response}
Documentation:

GET data from given URI, check status code and return response text. ${accept} is a Python object with headers to use. If ${normalize_json}, normalize as JSON text before returning.
Start / End / Elapsed: 20250222 23:56:42.664 / 20250222 23:56:42.684 / 00:00:00.020
00:00:00.000 KEYWORD BuiltIn . Log   ${uri}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250222 23:56:42.668 / 20250222 23:56:42.668 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${accept}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250222 23:56:42.669 / 20250222 23:56:42.669 / 00:00:00.000
00:00:00.012 IF """${http_timeout}""" == """${EMPTY}"""  
Start / End / Elapsed: 20250222 23:56:42.669 / 20250222 23:56:42.681 / 00:00:00.012
00:00:00.010 KEYWORD ${response} = RequestsLibrary . GET On Session   ${session} url=${uri} headers=${accept}
Documentation:

Sends a GET request on a previously created HTTP Session.
Start / End / Elapsed: 20250222 23:56:42.671 / 20250222 23:56:42.681 / 00:00:00.010
23:56:42.681 INFO GET Request : url=http://10.30.171.74:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore path_url=/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore headers={'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': 'Basic YWRtaW46YWRtaW4='} body=None
23:56:42.681 INFO GET Response : url=http://10.30.171.74:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore status=401, reason=Unauthorized headers={'Content-Length': '0'} body=None
23:56:42.681 FAIL HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore
00:00:00.000 ELSE  
Start / End / Elapsed: 20250222 23:56:42.681 / 20250222 23:56:42.681 / 00:00:00.000
00:00:00.000 KEYWORD TemplatedRequests . Check_Status_Code   ${response} log_response=${log_response}
Documentation:

Log response text, check status_code is one of allowed ones. In cases where this keyword is called in a WUKS it could end up logging tons of data and it may be desired to skip the logging by passing log_response=False, but by default it remains True.
Start / End / Elapsed: 20250222 23:56:42.683 / 20250222 23:56:42.683 / 00:00:00.000
00:00:00.000 IF not ${normalize_json}  
Start / End / Elapsed: 20250222 23:56:42.683 / 20250222 23:56:42.683 / 00:00:00.000
00:00:00.000 KEYWORD ${text_normalized} = norm_json . Normalize Json Text   ${response.text} jmes_path=${jmes_path} keys_with_volatiles=${keys_with_volatiles}
Documentation:

Attempt to return sorted indented JSON string.
Start / End / Elapsed: 20250222 23:56:42.683 / 20250222 23:56:42.683 / 00:00:00.000
00:00:00.000 RETURN   ${text_normalized}
Start / End / Elapsed: 20250222 23:56:42.684 / 20250222 23:56:42.684 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250222 23:56:42.684 / 20250222 23:56:42.684 / 00:00:00.000
00:00:00.000 KEYWORD ${conf_status} = ClusterManagement . ClusterManagement__Parse_Sync_Status   shard_manager_text=${conf_text}
Documentation:

Return sync status parsed out of given text. Called twice by Get_Sync_Status_Of_Member.
Start / End / Elapsed: 20250222 23:56:42.685 / 20250222 23:56:42.685 / 00:00:00.000
00:00:00.000 IF 'False' == ${conf_status}  
Start / End / Elapsed: 20250222 23:56:42.685 / 20250222 23:56:42.685 / 00:00:00.000
00:00:00.000 KEYWORD ${oper_text} = TemplatedRequests . Get_As_Json_From_Uri   uri=${JOLOKIA_OPER_SHARD_MANAGER_URI} session=${session}
Documentation:

Specify JSON headers and return Get_From_Uri normalized response text.
Start / End / Elapsed: 20250222 23:56:42.685 / 20250222 23:56:42.685 / 00:00:00.000
00:00:00.000 KEYWORD ${oper_status} = ClusterManagement . ClusterManagement__Parse_Sync_Status   shard_manager_text=${oper_text}
Documentation:

Return sync status parsed out of given text. Called twice by Get_Sync_Status_Of_Member.
Start / End / Elapsed: 20250222 23:56:42.686 / 20250222 23:56:42.686 / 00:00:00.000
00:00:00.000 RETURN   ${oper_status}
Start / End / Elapsed: 20250222 23:56:42.686 / 20250222 23:56:42.686 / 00:00:00.000
00:00:00.000 IF 'True' == '${status}'  
Start / End / Elapsed: 20250222 23:56:42.687 / 20250222 23:56:42.687 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Fail   Index ${index} has incorrect status: ${status}
Documentation:

Fails the test with the given message and optionally alters its tags.
Start / End / Elapsed: 20250222 23:56:42.687 / 20250222 23:56:42.687 / 00:00:00.000
00:00:00.001 IF ${verify_restconf}  
Start / End / Elapsed: 20250222 23:56:42.688 / 20250222 23:56:42.689 / 00:00:00.001
00:00:00.001 IF ${verify_system_status} and ("${service_list}" != "[[]]")  
Start / End / Elapsed: 20250222 23:56:42.689 / 20250222 23:56:42.690 / 00:00:00.001
23:56:42.690 FAIL Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore
00:00:02.021 TEARDOWN ClusterManagement . Run_Bash_Command_On_List_Or_All   command=netstat -pnatu | grep 2550
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250222 23:56:42.691 / 20250222 23:56:44.712 / 00:00:02.021
00:00:00.000 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 10s ClusterManagement.Run_Bash_Command_On_List_Or_All netstat -punta
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250222 23:56:44.712 / 20250222 23:56:44.712 / 00:00:00.000
00:00:00.000 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All  
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250222 23:56:44.713 / 20250222 23:56:44.713 / 00:00:00.000
00:00:00.001 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250222 23:56:44.713 / 20250222 23:56:44.714 / 00:00:00.001
00:00:00.000 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 10s ShardStability.Shards_Stability_Get_Details ${DEFAULT_SHARD_LIST} verify_restconf=True
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250222 23:56:44.714 / 20250222 23:56:44.714 / 00:00:00.000
00:04:32.947 SUITE Rpc Provider Partition And Heal
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal
Documentation:

DOMRpcBroker testing: RPC Provider Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This tests establishes that the RPC service operates correctly when faced with node failures. This suite supports more than three node cluster setup too.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot
Start / End / Elapsed: 20250222 23:56:44.716 / 20250223 00:01:17.663 / 00:04:32.947
Status: 7 tests total, 2 passed, 5 failed, 0 skipped
00:00:33.853 SETUP Setup_Kw  
Documentation:

Setup keyword. Create ${possible_constants} list with possible variables of remote constants.
Start / End / Elapsed: 20250222 23:56:44.793 / 20250222 23:57:18.646 / 00:00:33.853
00:00:00.000 TEARDOWN SSHLibrary . Close All Connections  
Documentation:

Closes all open connections.
Start / End / Elapsed: 20250223 00:01:17.663 / 20250223 00:01:17.663 / 00:00:00.000
00:00:21.500 TEST Register_Rpc_On_Two_Nodes
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal.Register_Rpc_On_Two_Nodes
Documentation:

Register rpc on two nodes of the odl cluster.
Tags: critical
Start / End / Elapsed: 20250222 23:57:18.647 / 20250222 23:57:40.147 / 00:00:21.500
Status: PASS
00:00:31.211 TEST Invoke_Rpc_On_Each_Node
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal.Invoke_Rpc_On_Each_Node
Documentation:

Invoke get-constant rpc on every node of the cluster. When requested on the node with local instance the local value is expected. If invoked on the node with no local instance, any remote value is expected.
Tags: critical
Start / End / Elapsed: 20250222 23:57:40.147 / 20250222 23:58:11.358 / 00:00:31.211
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_each_node&order=bug_status" Keyword 'Verify_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
00:00:20.358 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250222 23:57:40.147 / 20250222 23:58:00.505 / 00:00:20.358
00:00:10.842 KEYWORD DrbCommons . Verify_Constant_On_Active_Nodes  
Documentation:

Verify that the rpc response comes from the local node for every node in the list. As a workaround for Bug 8430, ${tolerance} can be set as duration (number of seconds) for WUKS.
Start / End / Elapsed: 20250222 23:58:00.506 / 20250222 23:58:11.348 / 00:00:10.842
00:00:10.840 KEYWORD BuiltIn . Run Keyword And Return If   ${tolerance} BuiltIn.Wait_Until_Keyword_Succeeds ${tolerance} 1s Verify_Constant_On_Active_Nodes tolerance=0
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250222 23:58:00.506 / 20250222 23:58:11.346 / 00:00:10.840
00:00:10.840 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${tolerance} 1s Verify_Constant_On_Active_Nodes tolerance=0
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250222 23:58:00.507 / 20250222 23:58:11.347 / 00:00:10.840
Message: 6 failing items removed using the --remove-keywords option.
00:00:00.699 KEYWORD DrbCommons . Verify_Constant_On_Active_Nodes   tolerance=0
Documentation:

Verify that the rpc response comes from the local node for every node in the list. As a workaround for Bug 8430, ${tolerance} can be set as duration (number of seconds) for WUKS.
Start / End / Elapsed: 20250222 23:58:10.648 / 20250222 23:58:11.347 / 00:00:00.699
00:00:00.000 KEYWORD BuiltIn . Run Keyword And Return If   ${tolerance} BuiltIn.Wait_Until_Keyword_Succeeds ${tolerance} 1s Verify_Constant_On_Active_Nodes tolerance=0
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250222 23:58:10.649 / 20250222 23:58:10.649 / 00:00:00.000
00:00:00.697 FOR ${index} IN @{active_indices}  
Start / End / Elapsed: 20250222 23:58:10.649 / 20250222 23:58:11.346 / 00:00:00.697
00:00:00.257 ITERATION ${index} = 1  
Start / End / Elapsed: 20250222 23:58:10.650 / 20250222 23:58:10.907 / 00:00:00.257
00:00:00.227 ITERATION ${index} = 2  
Start / End / Elapsed: 20250222 23:58:10.907 / 20250222 23:58:11.134 / 00:00:00.227
00:00:00.212 ITERATION ${index} = 3  
Start / End / Elapsed: 20250222 23:58:11.134 / 20250222 23:58:11.346 / 00:00:00.212
00:00:00.000 IF ${index} in ${registered_indices}  
Start / End / Elapsed: 20250222 23:58:11.134 / 20250222 23:58:11.134 / 00:00:00.000
00:00:00.212 ELSE  
Start / End / Elapsed: 20250222 23:58:11.135 / 20250222 23:58:11.347 / 00:00:00.212
00:00:00.212 KEYWORD DrbCommons . Verify_Constant_On_Unregistered_Node   ${index}
Documentation:

Verify that the response comes from other nodes with rpc registered. Verification passes for registered nodes too.
Start / End / Elapsed: 20250222 23:58:11.135 / 20250222 23:58:11.347 / 00:00:00.212
00:00:00.207 KEYWORD ${constant} = MdsalLowlevel . Get_Constant   ${member_index}
Documentation:

Invoke get-constant rpc on the requested member and return the registered constant unless explicit status code is expected. The ${explicit_status_codes} is a list of http status codes for which the rpc call is considered as passed and is used for calls with expected failures on odl's side, such as calling the rpc on isolated node etc.
Start / End / Elapsed: 20250222 23:58:11.139 / 20250222 23:58:11.346 / 00:00:00.207
00:00:00.001 KEYWORD ${session} = ClusterManagement . Resolve_Http_Session_For_Member   member_index=${member_index}
Documentation:

Return RequestsLibrary session alias pointing to node of given index.
Start / End / Elapsed: 20250222 23:58:11.140 / 20250222 23:58:11.141 / 00:00:00.001
00:00:00.205 KEYWORD ${text} = TemplatedRequests . Post_As_Xml_Templated   ${GET_CONSTANT_DIR} session=${session} explicit_status_codes=${explicit_status_codes}
Documentation:

Add arguments sensible for XML data, return Post_Templated response text. Optionally, verification against response.xml (no iteration) is called.
Start / End / Elapsed: 20250222 23:58:11.141 / 20250222 23:58:11.346 / 00:00:00.205
00:00:00.199 KEYWORD ${response_text} = TemplatedRequests . Post_Templated   folder=${folder} base_name=data extension=xml accept=${ACCEPT_XML} content_type=${HEADERS_XML} mapping=${mapping} session=${session} normalize_json=False endline=${\n} iterations=${iterations} iter_start=${iter_start} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes} http_timeout=${http_timeout} iter_j_offset=${iter_j_offset}
Documentation:

Resolve URI and data from folder, call Post_To_Uri, return response text.
Start / End / Elapsed: 20250222 23:58:11.146 / 20250222 23:58:11.345 / 00:00:00.199
00:00:00.062 KEYWORD ${uri} = TemplatedRequests . Resolve_Text_From_Template_Folder   folder=${folder} base_name=location extension=uri mapping=${mapping} percent_encode=True
Documentation:

Read a template from folder, strip endline, make changes according to mapping, return the result. If ${iterations} value is present, put text together from "prolog", "item" and "epilog" parts, where additional template variable ${i} goes from 1, by one times. Template variable ${j} is calculated as ${i} incremented by offset 0 ( j = i + iter_j_offset ) used to create non uniform data in order to be able to validate UPDATE operations. POST (as opposed to PUT) needs slightly different data, ${name_prefix} may be used to distinguish. (Actually, it is GET who formats data differently when URI is a top-level container.)
Start / End / Elapsed: 20250222 23:58:11.171 / 20250222 23:58:11.233 / 00:00:00.062
00:00:00.059 KEYWORD ${data} = TemplatedRequests . Resolve_Text_From_Template_Folder   folder=${folder} name_prefix=post_ base_name=${base_name} extension=${extension} mapping=${mapping} endline=${endline} iterations=${iterations} iter_start=${iter_start} iter_j_offset=${iter_j_offset}
Documentation:

Read a template from folder, strip endline, make changes according to mapping, return the result. If ${iterations} value is present, put text together from "prolog", "item" and "epilog" parts, where additional template variable ${i} goes from 1, by one times. Template variable ${j} is calculated as ${i} incremented by offset 0 ( j = i + iter_j_offset ) used to create non uniform data in order to be able to validate UPDATE operations. POST (as opposed to PUT) needs slightly different data, ${name_prefix} may be used to distinguish. (Actually, it is GET who formats data differently when URI is a top-level container.)
Start / End / Elapsed: 20250222 23:58:11.258 / 20250222 23:58:11.317 / 00:00:00.059
00:00:00.005 KEYWORD ${jmes_expression} = TemplatedRequests . Resolve_Jmes_Path   ${folder}
Documentation:

Reads JMES path from file /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/libraries/../variables/mdsal/lowlevelrpc/get_constant/jmespath.expr if the file exists and returns the JMES path. Empty string is returned otherwise.
Start / End / Elapsed: 20250222 23:58:11.317 / 20250222 23:58:11.322 / 00:00:00.005
00:00:00.022 KEYWORD ${response_text} = TemplatedRequests . Post_To_Uri   uri=${uri} data=${data} content_type=${content_type} accept=${accept} session=${session} jmes_path=${jmes_expression} normalize_json=${normalize_json} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes} http_timeout=${http_timeout}
Documentation:

POST data to given URI, check status code and return response text. ${content_type} and ${accept} are mandatory Python objects with headers to use. If ${normalize_json}, normalize text before returning.
Start / End / Elapsed: 20250222 23:58:11.323 / 20250222 23:58:11.345 / 00:00:00.022
00:00:00.000 KEYWORD BuiltIn . Log   ${uri}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250222 23:58:11.324 / 20250222 23:58:11.324 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${data}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250222 23:58:11.324 / 20250222 23:58:11.324 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${content_type}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250222 23:58:11.325 / 20250222 23:58:11.325 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${accept}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250222 23:58:11.325 / 20250222 23:58:11.325 / 00:00:00.000
00:00:00.003 KEYWORD ${headers} = TemplatedRequests . Join_Two_Headers   first=${content_type} second=${accept}
Documentation:

Take two dicts, join them, return result. Second argument values take precedence.
Start / End / Elapsed: 20250222 23:58:11.326 / 20250222 23:58:11.329 / 00:00:00.003
00:00:00.014 IF """${http_timeout}""" == """${EMPTY}"""  
Start / End / Elapsed: 20250222 23:58:11.329 / 20250222 23:58:11.343 / 00:00:00.014
00:00:00.011 KEYWORD ${response} = RequestsLibrary . POST On Session   ${session} ${uri} data=${data} headers=${headers}
Documentation:

Sends a POST request on a previously created HTTP Session.
Start / End / Elapsed: 20250222 23:58:11.331 / 20250222 23:58:11.342 / 00:00:00.011
23:58:11.342 INFO POST Request : url=http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant path_url=/rests/operations/odl-mdsal-lowlevel-target:get-constant headers={'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/xml', 'Connection': 'keep-alive', 'Content-Type': 'application/xml', 'Content-Length': '0', 'Authorization': 'Basic YWRtaW46YWRtaW4='} body=None
23:58:11.342 INFO POST Response : url=http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant status=401, reason=Unauthorized headers={'Content-Length': '0'} body=None
23:58:11.342 FAIL HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
00:00:00.001 ELSE  
Start / End / Elapsed: 20250222 23:58:11.343 / 20250222 23:58:11.344 / 00:00:00.001
00:00:00.000 KEYWORD TemplatedRequests . Check_Status_Code   ${response} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes}
Documentation:

Log response text, check status_code is one of allowed ones. In cases where this keyword is called in a WUKS it could end up logging tons of data and it may be desired to skip the logging by passing log_response=False, but by default it remains True.
Start / End / Elapsed: 20250222 23:58:11.344 / 20250222 23:58:11.344 / 00:00:00.000
00:00:00.000 IF not ${normalize_json}  
Start / End / Elapsed: 20250222 23:58:11.344 / 20250222 23:58:11.344 / 00:00:00.000
00:00:00.000 KEYWORD ${text_normalized} = norm_json . Normalize Json Text   ${response.text} jmes_path=${jmes_path}
Documentation:

Attempt to return sorted indented JSON string.
Start / End / Elapsed: 20250222 23:58:11.344 / 20250222 23:58:11.344 / 00:00:00.000
00:00:00.000 RETURN   ${text_normalized}
Start / End / Elapsed: 20250222 23:58:11.345 / 20250222 23:58:11.345 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250222 23:58:11.345 / 20250222 23:58:11.345 / 00:00:00.000
00:00:00.000 IF ${verify}  
Start / End / Elapsed: 20250222 23:58:11.345 / 20250222 23:58:11.345 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250222 23:58:11.345 / 20250222 23:58:11.345 / 00:00:00.000
00:00:00.000 IF """${explicit_status_codes}""" != """${NO_STATUS_CODES}"""  
Start / End / Elapsed: 20250222 23:58:11.345 / 20250222 23:58:11.345 / 00:00:00.000
00:00:00.000 KEYWORD ${xml} = XML . Parse Xml   ${text}
Documentation:

Parses the given XML file or string into an element structure.
Start / End / Elapsed: 20250222 23:58:11.346 / 20250222 23:58:11.346 / 00:00:00.000
00:00:00.000 KEYWORD ${constant} = XML . Get Element Text   ${xml} xpath=constant
Documentation:

Returns all text of the element, possibly whitespace normalized.
Start / End / Elapsed: 20250222 23:58:11.346 / 20250222 23:58:11.346 / 00:00:00.000
00:00:00.000 RETURN   ${constant}
Start / End / Elapsed: 20250222 23:58:11.346 / 20250222 23:58:11.346 / 00:00:00.000
00:00:00.000 KEYWORD Collections . List Should Contain Value   ${possible_constants} ${constant}
Documentation:

Fails if the value is not found from list.
Start / End / Elapsed: 20250222 23:58:11.346 / 20250222 23:58:11.346 / 00:00:00.000
00:00:00.000 RETURN   ${constant}
Start / End / Elapsed: 20250222 23:58:11.346 / 20250222 23:58:11.346 / 00:00:00.000
23:58:11.347 FAIL Keyword 'Verify_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
23:58:11.347 INFO Returning from the enclosing user keyword.
00:00:00.001 FOR ${index} IN @{active_indices}  
Start / End / Elapsed: 20250222 23:58:11.347 / 20250222 23:58:11.348 / 00:00:00.001
00:00:00.009 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs in case of failure.
Start / End / Elapsed: 20250222 23:58:11.348 / 20250222 23:58:11.357 / 00:00:00.009
00:00:20.464 TEST Isolate_One_Node
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal.Isolate_One_Node
Documentation:

Isolate one node with registered rpc. From the constant returned from the 3 node (with no rpc instance) an index of the node to be isolated is derived. And in the tc Invoke_Rpc_On_Remaining_Nodes a different constant is expected.
Tags: critical
Start / End / Elapsed: 20250222 23:58:11.358 / 20250222 23:58:31.822 / 00:00:20.464
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_partition_and_heal_isolate_one_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_partition_and_heal_isolate_one_node&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
00:00:20.281 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250222 23:58:11.358 / 20250222 23:58:31.639 / 00:00:20.281
00:00:00.170 KEYWORD ${isolated_idx} = DrbCommons . Get_Constant_Index_From_Node   ${TESTED_MEMBER_WITHOUT_RPC_IDX}
Documentation:

Ivoke get-constant rpc on given member index. Returns the index of the node where the constant came from.
Start / End / Elapsed: 20250222 23:58:31.639 / 20250222 23:58:31.809 / 00:00:00.170
00:00:00.165 KEYWORD ${constant} = MdsalLowlevel . Get_Constant   ${member_index}
Documentation:

Invoke get-constant rpc on the requested member and return the registered constant unless explicit status code is expected. The ${explicit_status_codes} is a list of http status codes for which the rpc call is considered as passed and is used for calls with expected failures on odl's side, such as calling the rpc on isolated node etc.
Start / End / Elapsed: 20250222 23:58:31.644 / 20250222 23:58:31.809 / 00:00:00.165
00:00:00.001 KEYWORD ${session} = ClusterManagement . Resolve_Http_Session_For_Member   member_index=${member_index}
Documentation:

Return RequestsLibrary session alias pointing to node of given index.
Start / End / Elapsed: 20250222 23:58:31.644 / 20250222 23:58:31.645 / 00:00:00.001
00:00:00.162 KEYWORD ${text} = TemplatedRequests . Post_As_Xml_Templated   ${GET_CONSTANT_DIR} session=${session} explicit_status_codes=${explicit_status_codes}
Documentation:

Add arguments sensible for XML data, return Post_Templated response text. Optionally, verification against response.xml (no iteration) is called.
Start / End / Elapsed: 20250222 23:58:31.645 / 20250222 23:58:31.807 / 00:00:00.162
00:00:00.155 KEYWORD ${response_text} = TemplatedRequests . Post_Templated   folder=${folder} base_name=data extension=xml accept=${ACCEPT_XML} content_type=${HEADERS_XML} mapping=${mapping} session=${session} normalize_json=False endline=${\n} iterations=${iterations} iter_start=${iter_start} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes} http_timeout=${http_timeout} iter_j_offset=${iter_j_offset}
Documentation:

Resolve URI and data from folder, call Post_To_Uri, return response text.
Start / End / Elapsed: 20250222 23:58:31.651 / 20250222 23:58:31.806 / 00:00:00.155
00:00:00.057 KEYWORD ${uri} = TemplatedRequests . Resolve_Text_From_Template_Folder   folder=${folder} base_name=location extension=uri mapping=${mapping} percent_encode=True
Documentation:

Read a template from folder, strip endline, make changes according to mapping, return the result. If ${iterations} value is present, put text together from "prolog", "item" and "epilog" parts, where additional template variable ${i} goes from 1, by one times. Template variable ${j} is calculated as ${i} incremented by offset 0 ( j = i + iter_j_offset ) used to create non uniform data in order to be able to validate UPDATE operations. POST (as opposed to PUT) needs slightly different data, ${name_prefix} may be used to distinguish. (Actually, it is GET who formats data differently when URI is a top-level container.)
Start / End / Elapsed: 20250222 23:58:31.661 / 20250222 23:58:31.718 / 00:00:00.057
00:00:00.052 KEYWORD ${data} = TemplatedRequests . Resolve_Text_From_Template_Folder   folder=${folder} name_prefix=post_ base_name=${base_name} extension=${extension} mapping=${mapping} endline=${endline} iterations=${iterations} iter_start=${iter_start} iter_j_offset=${iter_j_offset}
Documentation:

Read a template from folder, strip endline, make changes according to mapping, return the result. If ${iterations} value is present, put text together from "prolog", "item" and "epilog" parts, where additional template variable ${i} goes from 1, by one times. Template variable ${j} is calculated as ${i} incremented by offset 0 ( j = i + iter_j_offset ) used to create non uniform data in order to be able to validate UPDATE operations. POST (as opposed to PUT) needs slightly different data, ${name_prefix} may be used to distinguish. (Actually, it is GET who formats data differently when URI is a top-level container.)
Start / End / Elapsed: 20250222 23:58:31.728 / 20250222 23:58:31.780 / 00:00:00.052
00:00:00.004 KEYWORD ${jmes_expression} = TemplatedRequests . Resolve_Jmes_Path   ${folder}
Documentation:

Reads JMES path from file /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/libraries/../variables/mdsal/lowlevelrpc/get_constant/jmespath.expr if the file exists and returns the JMES path. Empty string is returned otherwise.
Start / End / Elapsed: 20250222 23:58:31.780 / 20250222 23:58:31.784 / 00:00:00.004
00:00:00.021 KEYWORD ${response_text} = TemplatedRequests . Post_To_Uri   uri=${uri} data=${data} content_type=${content_type} accept=${accept} session=${session} jmes_path=${jmes_expression} normalize_json=${normalize_json} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes} http_timeout=${http_timeout}
Documentation:

POST data to given URI, check status code and return response text. ${content_type} and ${accept} are mandatory Python objects with headers to use. If ${normalize_json}, normalize text before returning.
Start / End / Elapsed: 20250222 23:58:31.785 / 20250222 23:58:31.806 / 00:00:00.021
00:00:00.000 KEYWORD BuiltIn . Log   ${uri}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250222 23:58:31.785 / 20250222 23:58:31.785 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${data}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250222 23:58:31.786 / 20250222 23:58:31.786 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${content_type}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250222 23:58:31.786 / 20250222 23:58:31.786 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${accept}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250222 23:58:31.786 / 20250222 23:58:31.786 / 00:00:00.000
00:00:00.002 KEYWORD ${headers} = TemplatedRequests . Join_Two_Headers   first=${content_type} second=${accept}
Documentation:

Take two dicts, join them, return result. Second argument values take precedence.
Start / End / Elapsed: 20250222 23:58:31.787 / 20250222 23:58:31.789 / 00:00:00.002
00:00:00.014 IF """${http_timeout}""" == """${EMPTY}"""  
Start / End / Elapsed: 20250222 23:58:31.788 / 20250222 23:58:31.802 / 00:00:00.014
00:00:00.011 KEYWORD ${response} = RequestsLibrary . POST On Session   ${session} ${uri} data=${data} headers=${headers}
Documentation:

Sends a POST request on a previously created HTTP Session.
Start / End / Elapsed: 20250222 23:58:31.792 / 20250222 23:58:31.803 / 00:00:00.011
23:58:31.802 INFO POST Request : url=http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant path_url=/rests/operations/odl-mdsal-lowlevel-target:get-constant headers={'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/xml', 'Connection': 'keep-alive', 'Content-Type': 'application/xml', 'Content-Length': '0', 'Authorization': 'Basic YWRtaW46YWRtaW4='} body=None
23:58:31.802 INFO POST Response : url=http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant status=401, reason=Unauthorized headers={'Content-Length': '0'} body=None
23:58:31.802 FAIL HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
00:00:00.000 ELSE  
Start / End / Elapsed: 20250222 23:58:31.803 / 20250222 23:58:31.803 / 00:00:00.000
00:00:00.000 KEYWORD TemplatedRequests . Check_Status_Code   ${response} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes}
Documentation:

Log response text, check status_code is one of allowed ones. In cases where this keyword is called in a WUKS it could end up logging tons of data and it may be desired to skip the logging by passing log_response=False, but by default it remains True.
Start / End / Elapsed: 20250222 23:58:31.804 / 20250222 23:58:31.804 / 00:00:00.000
00:00:00.000 IF not ${normalize_json}  
Start / End / Elapsed: 20250222 23:58:31.804 / 20250222 23:58:31.804 / 00:00:00.000
00:00:00.000 KEYWORD ${text_normalized} = norm_json . Normalize Json Text   ${response.text} jmes_path=${jmes_path}
Documentation:

Attempt to return sorted indented JSON string.
Start / End / Elapsed: 20250222 23:58:31.805 / 20250222 23:58:31.805 / 00:00:00.000
00:00:00.000 RETURN   ${text_normalized}
Start / End / Elapsed: 20250222 23:58:31.805 / 20250222 23:58:31.805 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250222 23:58:31.806 / 20250222 23:58:31.806 / 00:00:00.000
00:00:00.000 IF ${verify}  
Start / End / Elapsed: 20250222 23:58:31.806 / 20250222 23:58:31.806 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250222 23:58:31.807 / 20250222 23:58:31.807 / 00:00:00.000
00:00:00.000 IF """${explicit_status_codes}""" != """${NO_STATUS_CODES}"""  
Start / End / Elapsed: 20250222 23:58:31.807 / 20250222 23:58:31.807 / 00:00:00.000
00:00:00.000 KEYWORD ${xml} = XML . Parse Xml   ${text}
Documentation:

Parses the given XML file or string into an element structure.
Start / End / Elapsed: 20250222 23:58:31.808 / 20250222 23:58:31.808 / 00:00:00.000
00:00:00.000 KEYWORD ${constant} = XML . Get Element Text   ${xml} xpath=constant
Documentation:

Returns all text of the element, possibly whitespace normalized.
Start / End / Elapsed: 20250222 23:58:31.808 / 20250222 23:58:31.808 / 00:00:00.000
00:00:00.000 RETURN   ${constant}
Start / End / Elapsed: 20250222 23:58:31.808 / 20250222 23:58:31.808 / 00:00:00.000
00:00:00.000 KEYWORD ${index} = String . Replace String   ${constant} ${CONSTANT_PREFIX} ${EMPTY}
Documentation:

Replaces search_for in the given string with replace_with.
Start / End / Elapsed: 20250222 23:58:31.809 / 20250222 23:58:31.809 / 00:00:00.000
00:00:00.000 KEYWORD ${index} = BuiltIn . Convert To Integer   ${index}
Documentation:

Converts the given item to an integer number.
Start / End / Elapsed: 20250222 23:58:31.809 / 20250222 23:58:31.809 / 00:00:00.000
00:00:00.000 RETURN   ${index}
Start / End / Elapsed: 20250222 23:58:31.809 / 20250222 23:58:31.809 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Set Suite Variable   ${isolated_idx}
Documentation:

Makes a variable available everywhere within the scope of the current suite.
Start / End / Elapsed: 20250222 23:58:31.810 / 20250222 23:58:31.810 / 00:00:00.000
00:00:00.000 KEYWORD DrbCommons . Isolate_Node   ${isolated_idx}
Documentation:

Isolate a member and update appropriate suite variables.
Start / End / Elapsed: 20250222 23:58:31.811 / 20250222 23:58:31.811 / 00:00:00.000
00:00:00.011 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs in case of failure.
Start / End / Elapsed: 20250222 23:58:31.811 / 20250222 23:58:31.822 / 00:00:00.011
00:01:33.467 TEST Invoke_Rpc_On_Nonisolated_Nodes
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal.Invoke_Rpc_On_Nonisolated_Nodes
Documentation:

Invoke rpc on non-islolated nodes.
Tags: critical
Start / End / Elapsed: 20250222 23:58:31.822 / 20250223 00:00:05.289 / 00:01:33.467
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_nonisolated_nodes" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_nonisolated_nodes&order=bug_status" Keyword 'DrbCommons.Verify_Constant_On_Active_Nodes' failed after retrying for 1 minute. The last error was: Keyword 'Verify_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
00:00:20.327 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250222 23:58:31.823 / 20250222 23:58:52.150 / 00:00:20.327
00:01:13.129 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 5s DrbCommons.Verify_Constant_On_Active_Nodes
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250222 23:58:52.150 / 20250223 00:00:05.279 / 00:01:13.129
Message: 4 failing items removed using the --remove-keywords option.
00:00:10.062 KEYWORD DrbCommons . Verify_Constant_On_Active_Nodes  
Documentation:

Verify that the rpc response comes from the local node for every node in the list. As a workaround for Bug 8430, ${tolerance} can be set as duration (number of seconds) for WUKS.
Start / End / Elapsed: 20250222 23:59:55.217 / 20250223 00:00:05.279 / 00:00:10.062
00:00:10.060 KEYWORD BuiltIn . Run Keyword And Return If   ${tolerance} BuiltIn.Wait_Until_Keyword_Succeeds ${tolerance} 1s Verify_Constant_On_Active_Nodes tolerance=0
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250222 23:59:55.218 / 20250223 00:00:05.278 / 00:00:10.060
00:00:10.059 KEYWORD BuiltIn . Wait Until Keyword Succeeds   ${tolerance} 1s Verify_Constant_On_Active_Nodes tolerance=0
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250222 23:59:55.219 / 20250223 00:00:05.278 / 00:00:10.059
Message: 6 failing items removed using the --remove-keywords option.
00:00:00.582 KEYWORD DrbCommons . Verify_Constant_On_Active_Nodes   tolerance=0
Documentation:

Verify that the rpc response comes from the local node for every node in the list. As a workaround for Bug 8430, ${tolerance} can be set as duration (number of seconds) for WUKS.
Start / End / Elapsed: 20250223 00:00:04.696 / 20250223 00:00:05.278 / 00:00:00.582
00:00:00.000 KEYWORD BuiltIn . Run Keyword And Return If   ${tolerance} BuiltIn.Wait_Until_Keyword_Succeeds ${tolerance} 1s Verify_Constant_On_Active_Nodes tolerance=0
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250223 00:00:04.697 / 20250223 00:00:04.697 / 00:00:00.000
00:00:00.581 FOR ${index} IN @{active_indices}  
Start / End / Elapsed: 20250223 00:00:04.697 / 20250223 00:00:05.278 / 00:00:00.581
00:00:00.200 ITERATION ${index} = 1  
Start / End / Elapsed: 20250223 00:00:04.698 / 20250223 00:00:04.898 / 00:00:00.200
00:00:00.191 ITERATION ${index} = 2  
Start / End / Elapsed: 20250223 00:00:04.897 / 20250223 00:00:05.088 / 00:00:00.191
00:00:00.189 ITERATION ${index} = 3  
Start / End / Elapsed: 20250223 00:00:05.088 / 20250223 00:00:05.277 / 00:00:00.189
00:00:00.000 IF ${index} in ${registered_indices}  
Start / End / Elapsed: 20250223 00:00:05.088 / 20250223 00:00:05.088 / 00:00:00.000
00:00:00.188 ELSE  
Start / End / Elapsed: 20250223 00:00:05.089 / 20250223 00:00:05.277 / 00:00:00.188
00:00:00.188 KEYWORD DrbCommons . Verify_Constant_On_Unregistered_Node   ${index}
Documentation:

Verify that the response comes from other nodes with rpc registered. Verification passes for registered nodes too.
Start / End / Elapsed: 20250223 00:00:05.089 / 20250223 00:00:05.277 / 00:00:00.188
00:00:00.183 KEYWORD ${constant} = MdsalLowlevel . Get_Constant   ${member_index}
Documentation:

Invoke get-constant rpc on the requested member and return the registered constant unless explicit status code is expected. The ${explicit_status_codes} is a list of http status codes for which the rpc call is considered as passed and is used for calls with expected failures on odl's side, such as calling the rpc on isolated node etc.
Start / End / Elapsed: 20250223 00:00:05.094 / 20250223 00:00:05.277 / 00:00:00.183
00:00:00.001 KEYWORD ${session} = ClusterManagement . Resolve_Http_Session_For_Member   member_index=${member_index}
Documentation:

Return RequestsLibrary session alias pointing to node of given index.
Start / End / Elapsed: 20250223 00:00:05.094 / 20250223 00:00:05.095 / 00:00:00.001
00:00:00.179 KEYWORD ${text} = TemplatedRequests . Post_As_Xml_Templated   ${GET_CONSTANT_DIR} session=${session} explicit_status_codes=${explicit_status_codes}
Documentation:

Add arguments sensible for XML data, return Post_Templated response text. Optionally, verification against response.xml (no iteration) is called.
Start / End / Elapsed: 20250223 00:00:05.095 / 20250223 00:00:05.274 / 00:00:00.179
00:00:00.171 KEYWORD ${response_text} = TemplatedRequests . Post_Templated   folder=${folder} base_name=data extension=xml accept=${ACCEPT_XML} content_type=${HEADERS_XML} mapping=${mapping} session=${session} normalize_json=False endline=${\n} iterations=${iterations} iter_start=${iter_start} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes} http_timeout=${http_timeout} iter_j_offset=${iter_j_offset}
Documentation:

Resolve URI and data from folder, call Post_To_Uri, return response text.
Start / End / Elapsed: 20250223 00:00:05.101 / 20250223 00:00:05.272 / 00:00:00.171
00:00:00.062 KEYWORD ${uri} = TemplatedRequests . Resolve_Text_From_Template_Folder   folder=${folder} base_name=location extension=uri mapping=${mapping} percent_encode=True
Documentation:

Read a template from folder, strip endline, make changes according to mapping, return the result. If ${iterations} value is present, put text together from "prolog", "item" and "epilog" parts, where additional template variable ${i} goes from 1, by one times. Template variable ${j} is calculated as ${i} incremented by offset 0 ( j = i + iter_j_offset ) used to create non uniform data in order to be able to validate UPDATE operations. POST (as opposed to PUT) needs slightly different data, ${name_prefix} may be used to distinguish. (Actually, it is GET who formats data differently when URI is a top-level container.)
Start / End / Elapsed: 20250223 00:00:05.113 / 20250223 00:00:05.175 / 00:00:00.062
00:00:00.056 KEYWORD ${data} = TemplatedRequests . Resolve_Text_From_Template_Folder   folder=${folder} name_prefix=post_ base_name=${base_name} extension=${extension} mapping=${mapping} endline=${endline} iterations=${iterations} iter_start=${iter_start} iter_j_offset=${iter_j_offset}
Documentation:

Read a template from folder, strip endline, make changes according to mapping, return the result. If ${iterations} value is present, put text together from "prolog", "item" and "epilog" parts, where additional template variable ${i} goes from 1, by one times. Template variable ${j} is calculated as ${i} incremented by offset 0 ( j = i + iter_j_offset ) used to create non uniform data in order to be able to validate UPDATE operations. POST (as opposed to PUT) needs slightly different data, ${name_prefix} may be used to distinguish. (Actually, it is GET who formats data differently when URI is a top-level container.)
Start / End / Elapsed: 20250223 00:00:05.185 / 20250223 00:00:05.241 / 00:00:00.056
00:00:00.005 KEYWORD ${jmes_expression} = TemplatedRequests . Resolve_Jmes_Path   ${folder}
Documentation:

Reads JMES path from file /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/libraries/../variables/mdsal/lowlevelrpc/get_constant/jmespath.expr if the file exists and returns the JMES path. Empty string is returned otherwise.
Start / End / Elapsed: 20250223 00:00:05.242 / 20250223 00:00:05.247 / 00:00:00.005
00:00:00.024 KEYWORD ${response_text} = TemplatedRequests . Post_To_Uri   uri=${uri} data=${data} content_type=${content_type} accept=${accept} session=${session} jmes_path=${jmes_expression} normalize_json=${normalize_json} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes} http_timeout=${http_timeout}
Documentation:

POST data to given URI, check status code and return response text. ${content_type} and ${accept} are mandatory Python objects with headers to use. If ${normalize_json}, normalize text before returning.
Start / End / Elapsed: 20250223 00:00:05.247 / 20250223 00:00:05.271 / 00:00:00.024
00:00:00.000 KEYWORD BuiltIn . Log   ${uri}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250223 00:00:05.249 / 20250223 00:00:05.249 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${data}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250223 00:00:05.249 / 20250223 00:00:05.249 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${content_type}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250223 00:00:05.249 / 20250223 00:00:05.249 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Log   ${accept}
Documentation:

Logs the given message with the given level.
Start / End / Elapsed: 20250223 00:00:05.250 / 20250223 00:00:05.250 / 00:00:00.000
00:00:00.002 KEYWORD ${headers} = TemplatedRequests . Join_Two_Headers   first=${content_type} second=${accept}
Documentation:

Take two dicts, join them, return result. Second argument values take precedence.
Start / End / Elapsed: 20250223 00:00:05.250 / 20250223 00:00:05.252 / 00:00:00.002
00:00:00.016 IF """${http_timeout}""" == """${EMPTY}"""  
Start / End / Elapsed: 20250223 00:00:05.252 / 20250223 00:00:05.268 / 00:00:00.016
00:00:00.013 KEYWORD ${response} = RequestsLibrary . POST On Session   ${session} ${uri} data=${data} headers=${headers}
Documentation:

Sends a POST request on a previously created HTTP Session.
Start / End / Elapsed: 20250223 00:00:05.254 / 20250223 00:00:05.267 / 00:00:00.013
00:00:05.266 INFO POST Request : url=http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant path_url=/rests/operations/odl-mdsal-lowlevel-target:get-constant headers={'User-Agent': 'python-requests/2.32.3', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/xml', 'Connection': 'keep-alive', 'Content-Type': 'application/xml', 'Content-Length': '0', 'Authorization': 'Basic YWRtaW46YWRtaW4='} body=None
00:00:05.266 INFO POST Response : url=http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant status=401, reason=Unauthorized headers={'Content-Length': '0'} body=None
00:00:05.266 FAIL HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
00:00:00.001 ELSE  
Start / End / Elapsed: 20250223 00:00:05.268 / 20250223 00:00:05.269 / 00:00:00.001
00:00:00.000 KEYWORD TemplatedRequests . Check_Status_Code   ${response} additional_allowed_status_codes=${additional_allowed_status_codes} explicit_status_codes=${explicit_status_codes}
Documentation:

Log response text, check status_code is one of allowed ones. In cases where this keyword is called in a WUKS it could end up logging tons of data and it may be desired to skip the logging by passing log_response=False, but by default it remains True.
Start / End / Elapsed: 20250223 00:00:05.269 / 20250223 00:00:05.269 / 00:00:00.000
00:00:00.000 IF not ${normalize_json}  
Start / End / Elapsed: 20250223 00:00:05.270 / 20250223 00:00:05.270 / 00:00:00.000
00:00:00.000 KEYWORD ${text_normalized} = norm_json . Normalize Json Text   ${response.text} jmes_path=${jmes_path}
Documentation:

Attempt to return sorted indented JSON string.
Start / End / Elapsed: 20250223 00:00:05.271 / 20250223 00:00:05.271 / 00:00:00.000
00:00:00.000 RETURN   ${text_normalized}
Start / End / Elapsed: 20250223 00:00:05.271 / 20250223 00:00:05.271 / 00:00:00.000
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250223 00:00:05.272 / 20250223 00:00:05.272 / 00:00:00.000
00:00:00.001 IF ${verify}  
Start / End / Elapsed: 20250223 00:00:05.273 / 20250223 00:00:05.274 / 00:00:00.001
00:00:00.000 RETURN   ${response_text}
Start / End / Elapsed: 20250223 00:00:05.274 / 20250223 00:00:05.274 / 00:00:00.000
00:00:00.000 IF """${explicit_status_codes}""" != """${NO_STATUS_CODES}"""  
Start / End / Elapsed: 20250223 00:00:05.275 / 20250223 00:00:05.275 / 00:00:00.000
00:00:00.000 KEYWORD ${xml} = XML . Parse Xml   ${text}
Documentation:

Parses the given XML file or string into an element structure.
Start / End / Elapsed: 20250223 00:00:05.275 / 20250223 00:00:05.275 / 00:00:00.000
00:00:00.000 KEYWORD ${constant} = XML . Get Element Text   ${xml} xpath=constant
Documentation:

Returns all text of the element, possibly whitespace normalized.
Start / End / Elapsed: 20250223 00:00:05.276 / 20250223 00:00:05.276 / 00:00:00.000
00:00:00.000 RETURN   ${constant}
Start / End / Elapsed: 20250223 00:00:05.276 / 20250223 00:00:05.276 / 00:00:00.000
00:00:00.000 KEYWORD Collections . List Should Contain Value   ${possible_constants} ${constant}
Documentation:

Fails if the value is not found from list.
Start / End / Elapsed: 20250223 00:00:05.277 / 20250223 00:00:05.277 / 00:00:00.000
00:00:00.000 RETURN   ${constant}
Start / End / Elapsed: 20250223 00:00:05.277 / 20250223 00:00:05.277 / 00:00:00.000
00:00:05.278 FAIL Keyword 'Verify_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
00:00:05.278 INFO Returning from the enclosing user keyword.
00:00:00.001 FOR ${index} IN @{active_indices}  
Start / End / Elapsed: 20250223 00:00:05.278 / 20250223 00:00:05.279 / 00:00:00.001
00:00:05.279 FAIL Keyword 'DrbCommons.Verify_Constant_On_Active_Nodes' failed after retrying for 1 minute. The last error was: Keyword 'Verify_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
00:00:00.010 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs in case of failure.
Start / End / Elapsed: 20250223 00:00:05.280 / 20250223 00:00:05.290 / 00:00:00.010
00:00:20.268 TEST Rejoin_Isolated_Member
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal.Rejoin_Isolated_Member
Documentation:

Rejoin isolated node
Tags: critical
Start / End / Elapsed: 20250223 00:00:05.289 / 20250223 00:00:25.557 / 00:00:20.268
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_partition_and_heal_rejoin_isolated_member" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_partition_and_heal_rejoin_isolated_member&order=bug_status" Variable '${isolated_idx}' not found.
00:00:20.253 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250223 00:00:05.290 / 20250223 00:00:25.543 / 00:00:20.253
00:00:00.004 KEYWORD DrbCommons . Rejoin_Node   ${isolated_idx}
Documentation:

Rejoin a member and update appropriate suite variables.
Start / End / Elapsed: 20250223 00:00:25.543 / 20250223 00:00:25.547 / 00:00:00.004
00:00:25.547 FAIL Variable '${isolated_idx}' not found.
00:00:00.009 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs in case of failure.
Start / End / Elapsed: 20250223 00:00:25.547 / 20250223 00:00:25.556 / 00:00:00.009
00:00:31.469 TEST Invoke_Rpc_On_Each_Node_Again
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal.Invoke_Rpc_On_Each_Node_Again
Documentation:

Invoke rpc get-constant on every node. When requested on the node with local instance the local value is expected. If invoked on the node with no local instance, any remote value is expected.
Tags: critical
Start / End / Elapsed: 20250223 00:00:25.557 / 20250223 00:00:57.026 / 00:00:31.469
Status: FAIL
Message: ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_each_node_again" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_each_node_again&order=bug_status" Keyword 'Verify_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
00:00:20.283 SETUP SetupUtils . Setup_Test_With_Logging_And_Without_Fast_Failing  
Documentation:

Test case setup which explicitly ignores previous failure and logs test case name to Karaf log. Needed if the recommended default is to be overriden.
Start / End / Elapsed: 20250223 00:00:25.558 / 20250223 00:00:45.841 / 00:00:20.283
00:00:11.175 KEYWORD WaitForFailure . Verify_Keyword_Does_Not_Fail_Within_Timeout   20s 3s DrbCommons.Verify_Constant_On_Active_Nodes
Documentation:

Some positive checks report false success for a short time. This keyword verifies no failure does happen within timeout period. This implementation needs more complicated logic than, Verify_Keyword_Never_Passes_Within_Timeout, so use that keyword in case you have a negative check handy.
Start / End / Elapsed: 20250223 00:00:45.841 / 20250223 00:00:57.016 / 00:00:11.175
00:00:11.174 KEYWORD BuiltIn . Run Keyword And Return   Invert_Failure Confirm_Keyword_Fails_Within_Timeout ${timeout} ${refresh} @{cell_list}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250223 00:00:45.841 / 20250223 00:00:57.015 / 00:00:11.174
00:00:11.174 KEYWORD WaitForFailure . Invert_Failure   Confirm_Keyword_Fails_Within_Timeout ${timeout} ${refresh} @{cell_list}
Documentation:

The response of Keyword execution is either a return value or a failure message. This keyword calls the argument keyword and returns its failure message string, or fails with its return value converted to string.
Start / End / Elapsed: 20250223 00:00:45.842 / 20250223 00:00:57.016 / 00:00:11.174
00:00:11.172 KEYWORD ${status} ${output} = BuiltIn . Run Keyword And Ignore Error   @{cell_sequence}
Documentation:

Runs the given keyword with the given arguments and ignores possible error.
Start / End / Elapsed: 20250223 00:00:45.842 / 20250223 00:00:57.014 / 00:00:11.172
00:00:00.000 IF "${status}" != "PASS"  
Start / End / Elapsed: 20250223 00:00:57.014 / 20250223 00:00:57.014 / 00:00:00.000
00:00:00.000 KEYWORD ${output} = BuiltIn . Convert To String   ${output}
Documentation:

Converts the given item to a Unicode string.
Start / End / Elapsed: 20250223 00:00:57.014 / 20250223 00:00:57.014 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Fail   ${output}
Documentation:

Fails the test with the given message and optionally alters its tags.
Start / End / Elapsed: 20250223 00:00:57.015 / 20250223 00:00:57.015 / 00:00:00.000
00:00:57.015 FAIL Keyword 'Verify_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/rests/operations/odl-mdsal-lowlevel-target:get-constant
00:00:57.015 INFO Returning from the enclosing user keyword.
00:00:00.010 TEARDOWN SetupUtils . Teardown_Test_Show_Bugs_If_Test_Failed  
Documentation:

Test case teardown. Show linked bugs in case of failure.
Start / End / Elapsed: 20250223 00:00:57.016 / 20250223 00:00:57.026 / 00:00:00.010
00:00:20.635 TEST Unregister_Rpc_On_Each_Node
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal.Unregister_Rpc_On_Each_Node
Documentation:

Inregister rpc on both nodes.
Tags: critical
Start / End / Elapsed: 20250223 00:00:57.027 / 20250223 00:01:17.662 / 00:00:20.635
Status: PASS
00:13:09.253 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 00:01:17.664 / 20250223 00:14:26.917 / 00:13:09.253
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:32.787 SETUP SetupUtils . Setup_Utils_For_Setup_And_Teardown   http_timeout=125
Documentation:

Prepare both FailFast and Karaf logging, log that the caller suite has started. This keyword is to be used in suite setup, and without it the other keywords may not work properly.
Start / End / Elapsed: 20250223 00:01:17.756 / 20250223 00:01:50.543 / 00:00:32.787
00:00:00.000 TEARDOWN SSHLibrary . Close All Connections  
Documentation:

Closes all open connections.
Start / End / Elapsed: 20250223 00:14:26.917 / 20250223 00:14:26.917 / 00:00:00.000
00:06:10.044 TEST Stop_All_Members
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Stop_All_Members
Documentation:

Stop every odl node.
Tags: critical
Start / End / Elapsed: 20250223 00:01:50.543 / 20250223 00:08:00.587 / 00:06:10.044
Status: FAIL
Message: Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1
00:06:10.044 KEYWORD ClusterManagement . Stop_Members_From_List_Or_All  
Documentation:

If the list is empty, stops all ODL instances. Otherwise stop members based on ${stop_index_list} If ${confirm} is True, verify stopped instances are not there anymore. The KW will return a list of available members: ${updated index_list}=${original_index_list}-${member_index_list}
Start / End / Elapsed: 20250223 00:01:50.544 / 20250223 00:08:00.588 / 00:06:10.044
00:00:00.001 KEYWORD ${stop_index_list} = ClusterManagement . List_Indices_Or_All   given_list=${member_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250223 00:01:50.548 / 20250223 00:01:50.549 / 00:00:00.001
00:00:00.001 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All   given_list=${original_index_list}
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250223 00:01:50.550 / 20250223 00:01:50.551 / 00:00:00.001
00:00:02.207 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=${NODE_STOP_COMMAND} member_index_list=${member_index_list}
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250223 00:01:50.551 / 20250223 00:01:52.758 / 00:00:02.207
00:00:00.000 KEYWORD ${updated_index_list} = BuiltIn . Create List   @{index_list}
Documentation:

Returns a list containing given items.
Start / End / Elapsed: 20250223 00:01:52.759 / 20250223 00:01:52.759 / 00:00:00.000
00:00:00.000 KEYWORD Collections . Remove Values From List   ${updated_index_list} @{stop_index_list}
Documentation:

Removes all occurrences of given values from list.
Start / End / Elapsed: 20250223 00:01:52.759 / 20250223 00:01:52.759 / 00:00:00.000
00:00:00.000 IF not ${confirm}  
Start / End / Elapsed: 20250223 00:01:52.759 / 20250223 00:01:52.759 / 00:00:00.000
00:06:07.828 FOR ${index} IN @{stop_index_list}  
Start / End / Elapsed: 20250223 00:01:52.760 / 20250223 00:08:00.588 / 00:06:07.828
00:00:05.442 ITERATION ${index} = 1  
Start / End / Elapsed: 20250223 00:01:52.760 / 20250223 00:01:58.202 / 00:00:05.442
00:00:00.476 ITERATION ${index} = 2  
Start / End / Elapsed: 20250223 00:01:58.202 / 20250223 00:01:58.678 / 00:00:00.476
00:06:01.909 ITERATION ${index} = 3  
Start / End / Elapsed: 20250223 00:01:58.678 / 20250223 00:08:00.587 / 00:06:01.909
00:00:00.000 KEYWORD ClusterManagement . Run_Bash_Command_On_List_Or_All   command=netstat -pnatu | grep 2550
Documentation:

Cycle through indices (or all), run command on each.
Start / End / Elapsed: 20250223 00:08:00.587 / 20250223 00:08:00.587 / 00:00:00.000
00:00:00.000 RETURN   ${updated_index_list}
Start / End / Elapsed: 20250223 00:08:00.588 / 20250223 00:08:00.588 / 00:00:00.000
00:00:13.118 TEST Unset_Tell_Based_Protocol_Usage
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Unset_Tell_Based_Protocol_Usage
Documentation:

Comment out the flag usage in config file. Also clean most data except data/log/.
Tags: critical
Start / End / Elapsed: 20250223 00:08:00.588 / 20250223 00:08:13.706 / 00:00:13.118
Status: PASS
00:06:13.211 TEST Start_All_And_Sync
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Start_All_And_Sync
Documentation:

Start each member and wait for sync.
Tags: critical
Start / End / Elapsed: 20250223 00:08:13.706 / 20250223 00:14:26.917 / 00:06:13.211
Status: FAIL
Message: Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.74:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore
00:03:20.299 SUITE Action Provider Precedence
Full Name: controller-clustering-ask.txt.Action Provider Precedence
Documentation:

DOMRpcBroker testing: Action Provider Precedence

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

The aim is to establish that remote action implementations have lower priority than local ones, which is to say that any movement of actions on remote nodes does not affect routing as long as a local implementation is available.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot
Start / End / Elapsed: 20250223 00:14:26.918 / 20250223 00:17:47.217 / 00:03:20.299
Status: 8 tests total, 4 passed, 4 failed, 0 skipped
00:13:09.561 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 00:17:47.218 / 20250223 00:30:56.779 / 00:13:09.561
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:04:33.553 SUITE Action Provider Partition And Heal
Full Name: controller-clustering-ask.txt.Action Provider Partition And Heal
Documentation:

DOMRpcBroker testing: RPC Action Provider Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This tests establishes that the RPC service for actions operates correctly when faced with node failures. This suite supports more than three node cluster setup too.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot
Start / End / Elapsed: 20250223 00:30:56.780 / 20250223 00:35:30.333 / 00:04:33.553
Status: 7 tests total, 2 passed, 5 failed, 0 skipped
00:13:05.538 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 00:35:30.334 / 20250223 00:48:35.872 / 00:13:05.538
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:02:38.417 SUITE Master Stability
Full Name: controller-clustering-ask.txt.Master Stability
Documentation:

Cluster Singleton testing: Master Stability

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Cluster Singleton service is designed to ensure that only one instance of an application is registered globally in the cluster. The goal is to establish the service operates correctly in face of application registration changing without moving the active instance.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/cluster_singleton/master_stability.robot
Start / End / Elapsed: 20250223 00:48:35.872 / 20250223 00:51:14.289 / 00:02:38.417
Status: 6 tests total, 0 passed, 6 failed, 0 skipped
00:13:08.947 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 00:51:14.290 / 20250223 01:04:23.237 / 00:13:08.947
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:02:38.030 SUITE Partition And Heal
Full Name: controller-clustering-ask.txt.Partition And Heal
Documentation:

Cluster Singleton testing: Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Cluster Singleton service is designed to ensure that only one instance of an application is registered globally in the cluster. The goal is to establish the service operates correctly in face of node failures.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot
Start / End / Elapsed: 20250223 01:04:23.238 / 20250223 01:07:01.268 / 00:02:38.030
Status: 6 tests total, 0 passed, 6 failed, 0 skipped
00:13:07.689 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 01:07:01.269 / 20250223 01:20:08.958 / 00:13:07.689
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:02:36.880 SUITE Chasing The Leader
Full Name: controller-clustering-ask.txt.Chasing The Leader
Documentation:

Cluster Singleton testing: Chasing the Leader

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test aims to establish the service operates correctly when faced with rapid application transitions without having a stabilized application.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot
Start / End / Elapsed: 20250223 01:20:08.959 / 20250223 01:22:45.839 / 00:02:36.880
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:13:07.748 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 01:22:45.839 / 20250223 01:35:53.587 / 00:13:07.748
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:44.767 SUITE Global Rpc Kill
Full Name: controller-clustering-ask.txt.Global Rpc Kill
Documentation:

Controller functional HA testing of global singleton rpcs.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot
Start / End / Elapsed: 20250223 01:35:53.588 / 20250223 01:36:38.355 / 00:00:44.767
Status: 11 tests total, 0 passed, 11 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:06.536 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 01:36:38.355 / 20250223 01:49:44.891 / 00:13:06.536
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:44.266 SUITE Global Rpc Freeze
Full Name: controller-clustering-ask.txt.Global Rpc Freeze
Documentation:

Controller functional HA testing of global singleton rpcs if jvm frozen.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot
Start / End / Elapsed: 20250223 01:49:44.892 / 20250223 01:50:29.158 / 00:00:44.266
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:05.452 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 01:50:29.158 / 20250223 02:03:34.610 / 00:13:05.452
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:44.361 SUITE Global Rpc Isolate
Full Name: controller-clustering-ask.txt.Global Rpc Isolate
Documentation:

Controller functional HA testing of global singleton rpcs.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot
Start / End / Elapsed: 20250223 02:03:34.611 / 20250223 02:04:18.972 / 00:00:44.361
Status: 11 tests total, 0 passed, 11 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:03.358 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 02:04:18.973 / 20250223 02:17:22.331 / 00:13:03.358
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:35.045 SUITE Carpeople Crud
Full Name: controller-clustering-ask.txt.Carpeople Crud
Documentation:

Suite for performing basic car/people CRUD operations on leaders and followers.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

More precisely, Update operation is not executed, but some operations are using specific RPCs which goes beyond "basic CRUD".

Cars are added by one big PUT to datastore on car Leader. People are added in a loop with add-person RPC on a people Follower. Cars are bought by chunks on each member, by loop with buy-car RPC.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot
Start / End / Elapsed: 20250223 02:17:22.332 / 20250223 02:17:57.377 / 00:00:35.045
Status: 13 tests total, 0 passed, 13 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.170.51:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:33.391 SUITE Car Failover Crud
Full Name: controller-clustering-ask.txt.Car Failover Crud
Documentation:

Suite mixing basic operations with restart of car Leader.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test stops the current leader of the "car" shard and then executes CRD operations on the new leader and a new follower. The stopped member is brought back. This suite uses 3 different car sets, same size but different starting ID.

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot
Start / End / Elapsed: 20250223 02:17:57.378 / 20250223 02:18:30.769 / 00:00:33.391
Status: 16 tests total, 0 passed, 16 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.31:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:33.407 SUITE Car Persistence Recovery
Full Name: controller-clustering-ask.txt.Car Persistence Recovery
Documentation:

This test restarts all controllers to verify recovery of car data from persistence.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot
Start / End / Elapsed: 20250223 02:18:30.769 / 20250223 02:19:04.176 / 00:00:33.407
Status: 7 tests total, 0 passed, 7 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.31:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:32.640 SUITE Buycar Failover
Full Name: controller-clustering-ask.txt.Buycar Failover
Documentation:

This test focuses on testing buy-car RPC over 3 Leader reboots.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

All purchases are against the same node, which is the first one to get rebooted.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot
Start / End / Elapsed: 20250223 02:19:04.177 / 20250223 02:19:36.817 / 00:00:32.640
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.31:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:34.512 SUITE Entity Isolate
Full Name: controller-clustering-ask.txt.Entity Isolate
Documentation:

Suite for performing member isolation and rejoin, we do with entity-ownership leader.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot
Start / End / Elapsed: 20250223 02:19:36.818 / 20250223 02:20:11.330 / 00:00:34.512
Status: 5 tests total, 0 passed, 5 failed, 0 skipped
Message: Suite setup failed: Evaluating expression 'json.loads(\'\'\'{\n "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "error_type": "javax.management.InstanceNotFoundException",\n "request": {\n "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "type": "read"\n },\n "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1358)\\n\\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:921)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\\n\\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\\n\\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:233)\\n\\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:245)\\n\\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:221)\\n\\tat org.jolokia.handler.ReadHandler.f... [ Message content over the limit has been removed. ] ...clipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\\n\\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\\n\\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\\n\\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\\n\\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\\n\\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\\n\\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\\n\\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\\n\\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\\n\\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\\n\\tat java.base/java.lang.Thread.run(Thread.java:840)\\n",\n "status": 404\n}\n\'\'\')' failed: JSONDecodeError: Invalid control character at: line 8 column 190 (char 619)
00:00:32.648 SUITE Buycar Failover Isolation
Full Name: controller-clustering-ask.txt.Buycar Failover Isolation
Documentation:

This test focuses on testing buy-car RPC over the Leader isolation.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

All purchases are against the same node, which is the first one to get isolated.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot
Start / End / Elapsed: 20250223 02:20:11.331 / 20250223 02:20:43.979 / 00:00:32.648
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.31:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:33.372 SUITE Car Failover Crud Isolation
Full Name: controller-clustering-ask.txt.Car Failover Crud Isolation
Documentation:

Suite mixing basic operations with isolation of car Leader.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test isolates the current leader of the "car" shard and then executes CRD operations on the new leader and a new follower. The isolated member is brought back. This suite uses 3 different car sets, same size but different starting ID.

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot
Start / End / Elapsed: 20250223 02:20:43.979 / 20250223 02:21:17.351 / 00:00:33.372
Status: 16 tests total, 0 passed, 16 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.31:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:32.613 SUITE Car Outage Corners
Full Name: controller-clustering-ask.txt.Car Outage Corners
Documentation:

Cluster suite for testing minimal and sum-minimal member population behavior.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test stops majority of the followers and verifies car addition is not possible, then resumes single follower (first from original list) and checks that addition works. Then remaining members are brought up. Leader member is always up and assumed to remain Leading during the whole suite run.

TODO: Use initial data to check more operations. TODO: Perhaps merge with car_failover_crud suite.

Other modules and Shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-calcium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot
Start / End / Elapsed: 20250223 02:21:17.351 / 20250223 02:21:49.964 / 00:00:32.613
Status: 8 tests total, 0 passed, 8 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.171.31:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig