Test Statistics

Total Statistics
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
All Tests
19529166002:27:26
Statistics by Tag
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
carpeople
85085000:00:00
clustering
85085000:00:00
critical
19529166002:27:26
Statistics by Suite
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
controller-clustering-ask.txt
19529166002:42:31
controller-clustering-ask . txt . Restart Odl With Tell Based False
321000:05:41
controller-clustering-ask . txt . Rpc Provider Precedence
880000:00:11
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:12:38
controller-clustering-ask . txt . Rpc Provider Partition And Heal
725000:04:21
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:08
controller-clustering-ask . txt . Action Provider Precedence
844000:03:20
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:01
controller-clustering-ask . txt . Action Provider Partition And Heal
725000:04:20
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:09
controller-clustering-ask . txt . Master Stability
606000:02:37
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:08
controller-clustering-ask . txt . Partition And Heal
606000:02:38
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:02
controller-clustering-ask . txt . Chasing The Leader
312000:02:35
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:02
controller-clustering-ask . txt . Global Rpc Kill
11011000:00:43
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:01
controller-clustering-ask . txt . Global Rpc Freeze
10010000:00:44
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:04
controller-clustering-ask . txt . Global Rpc Isolate
11011000:00:44
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:12:58
controller-clustering-ask . txt . Carpeople Crud
13013000:00:34
controller-clustering-ask . txt . Car Failover Crud
16016000:00:32
controller-clustering-ask . txt . Car Persistence Recovery
707000:00:33
controller-clustering-ask . txt . Buycar Failover
10010000:00:33
controller-clustering-ask . txt . Entity Isolate
505000:00:34
controller-clustering-ask . txt . Buycar Failover Isolation
10010000:00:32
controller-clustering-ask . txt . Car Failover Crud Isolation
16016000:00:33
controller-clustering-ask . txt . Car Outage Corners
808000:00:33

Test Execution Log

02:42:31.136 SUITE controller-clustering-ask.txt
Full Name: controller-clustering-ask.txt
Start / End / Elapsed: 20250223 23:08:21.625 / 20250224 01:50:52.761 / 02:42:31.136
Status: 195 tests total, 29 passed, 166 failed, 0 skipped
00:05:40.868 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 23:08:21.647 / 20250223 23:14:02.515 / 00:05:40.868
Status: 3 tests total, 2 passed, 1 failed, 0 skipped
00:00:04.894 SETUP SetupUtils . Setup_Utils_For_Setup_And_Teardown   http_timeout=125
Documentation:

Prepare both FailFast and Karaf logging, log that the caller suite has started. This keyword is to be used in suite setup, and without it the other keywords may not work properly.
Start / End / Elapsed: 20250223 23:08:22.176 / 20250223 23:08:27.070 / 00:00:04.894
00:00:00.000 TEARDOWN SSHLibrary . Close All Connections  
Documentation:

Closes all open connections.
Start / End / Elapsed: 20250223 23:14:02.514 / 20250223 23:14:02.514 / 00:00:00.000
00:04:13.444 TEST Stop_All_Members
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Stop_All_Members
Documentation:

Stop every odl node.
Tags: critical
Start / End / Elapsed: 20250223 23:08:27.069 / 20250223 23:12:40.513 / 00:04:13.444
Status: PASS
00:00:10.224 TEST Unset_Tell_Based_Protocol_Usage
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Unset_Tell_Based_Protocol_Usage
Documentation:

Comment out the flag usage in config file. Also clean most data except data/log/.
Tags: critical
Start / End / Elapsed: 20250223 23:12:40.513 / 20250223 23:12:50.737 / 00:00:10.224
Status: PASS
00:01:11.777 TEST Start_All_And_Sync
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Start_All_And_Sync
Documentation:

Start each member and wait for sync.
Tags: critical
Start / End / Elapsed: 20250223 23:12:50.737 / 20250223 23:14:02.514 / 00:01:11.777
Status: FAIL
Message: Got rc: 1 or stderr was not empty: sudo: netstat: command not found
00:01:09.998 KEYWORD ClusterManagement . Start_Members_From_List_Or_All  
Documentation:

If the list is empty, start all cluster members. Otherwise, start members based on present indices. If ${wait_for_sync}, wait for cluster sync on listed members. Optionally karaf_home can be overriden. Optionally specific JAVA_HOME is used for starting. Garbage collection is unconditionally logged to files. TODO: Make that reasonable conditional?
Start / End / Elapsed: 20250223 23:12:50.741 / 20250223 23:14:00.739 / 00:01:09.998
00:00:01.293 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 10s ClusterManagement.Run_Bash_Command_On_List_Or_All netstat -punta
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250223 23:14:00.740 / 20250223 23:14:02.033 / 00:00:01.293
00:00:00.003 KEYWORD ${index_list} = ClusterManagement . List_Indices_Or_All  
Documentation:

Utility to allow ${EMPTY} as default argument value, as the internal list is computed at runtime. This keyword always returns a (shallow) copy of given or default list, so operations with the returned list should not affect other lists. Also note that this keyword does not consider empty list to be ${EMPTY}.
Start / End / Elapsed: 20250223 23:14:02.033 / 20250223 23:14:02.036 / 00:00:00.003
00:00:00.478 FOR ${index} IN @{index_list}  
Start / End / Elapsed: 20250223 23:14:02.036 / 20250223 23:14:02.514 / 00:00:00.478
00:00:00.477 ITERATION ${index} = 1  
Start / End / Elapsed: 20250223 23:14:02.036 / 20250223 23:14:02.513 / 00:00:00.477
00:00:00.477 KEYWORD ${output} = ClusterManagement . Check_Bash_Command_On_Member   command=sudo netstat -punta | grep 2550 | grep LISTEN member_index=${index}
Documentation:

Open SSH session, call SSHKeywords.Execute_Command_Passes, close session, restore previously active session and return output.
Start / End / Elapsed: 20250223 23:14:02.036 / 20250223 23:14:02.513 / 00:00:00.477
00:00:00.476 KEYWORD BuiltIn . Run Keyword And Return   SSHKeywords.Run_Keyword_Preserve_Connection Check_Unsafely_Bash_Command_On_Member ${command} ${member_index} return_success_only=${return_success_only} log_on_success=${log_on_success} log_on_failure=${log_on_failure} stderr_must_be_empty=${stderr_must_be_empty}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250223 23:14:02.037 / 20250223 23:14:02.513 / 00:00:00.476
00:00:00.476 KEYWORD SSHKeywords . Run_Keyword_Preserve_Connection   Check_Unsafely_Bash_Command_On_Member ${command} ${member_index} return_success_only=${return_success_only} log_on_success=${log_on_success} log_on_failure=${log_on_failure} stderr_must_be_empty=${stderr_must_be_empty}
Documentation:

Store current connection index, run keyword returning its result, restore connection in teardown. Note that in order to avoid "got positional argument after named arguments", it is safer to use positional (not named) arguments on call.
Start / End / Elapsed: 20250223 23:14:02.037 / 20250223 23:14:02.513 / 00:00:00.476
00:00:00.000 KEYWORD ${current_connection} = SSHLibrary . Get Connection  
Documentation:

Returns information about the connection.
Start / End / Elapsed: 20250223 23:14:02.037 / 20250223 23:14:02.037 / 00:00:00.000
00:00:00.471 KEYWORD BuiltIn . Run Keyword And Return   ${keyword_name} @{args} &{kwargs}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250223 23:14:02.038 / 20250223 23:14:02.509 / 00:00:00.471
00:00:00.471 KEYWORD ClusterManagement . Check_Unsafely_Bash_Command_On_Member   @{args} &{kwargs}
Documentation:

Obtain Ip address, open session, call SSHKeywords.Execute_Command_Passes, close session and return output. This affects which SSH session is active.
Start / End / Elapsed: 20250223 23:14:02.038 / 20250223 23:14:02.509 / 00:00:00.471
00:00:00.001 KEYWORD ${member_ip} = ClusterManagement . Resolve_IP_Address_For_Member   ${member_index}
Documentation:

Return node IP address of given index.
Start / End / Elapsed: 20250223 23:14:02.039 / 20250223 23:14:02.040 / 00:00:00.001
00:00:00.469 KEYWORD BuiltIn . Run Keyword And Return   SSHKeywords.Run_Unsafely_Keyword_Over_Temporary_Odl_Session ${member_ip} Execute_Command_Passes ${command} return_success_only=${return_success_only} log_on_success=${log_on_success} log_on_failure=${log_on_failure} stderr_must_be_empty=${stderr_must_be_empty}
Documentation:

Runs the specified keyword and returns from the enclosing user keyword.
Start / End / Elapsed: 20250223 23:14:02.039 / 20250223 23:14:02.508 / 00:00:00.469
00:00:00.469 KEYWORD SSHKeywords . Run_Unsafely_Keyword_Over_Temporary_Odl_Session   ${member_ip} Execute_Command_Passes ${command} return_success_only=${return_success_only} log_on_success=${log_on_success} log_on_failure=${log_on_failure} stderr_must_be_empty=${stderr_must_be_empty}
Documentation:

Open connection to given IP address, run keyword, close connection, return result. This is unsafe in the sense that previously active session will be switched out off, but safe in the sense only the temporary connection is closed.
Start / End / Elapsed: 20250223 23:14:02.040 / 20250223 23:14:02.509 / 00:00:00.469
00:00:00.429 KEYWORD SSHKeywords . Open_Connection_To_ODL_System   ${ip_address}
Documentation:

Open a connection to the ODL system at 10.30.170.174 and return its identifier.
Start / End / Elapsed: 20250223 23:14:02.041 / 20250223 23:14:02.470 / 00:00:00.429
00:00:00.037 KEYWORD ${status} ${result} = BuiltIn . Run Keyword And Ignore Error   ${keyword_name} @{args} &{kwargs}
Documentation:

Runs the given keyword with the given arguments and ignores possible error.
Start / End / Elapsed: 20250223 23:14:02.470 / 20250223 23:14:02.507 / 00:00:00.037
00:00:00.000 KEYWORD SSHLibrary . Close Connection  
Documentation:

Closes the current connection.
Start / End / Elapsed: 20250223 23:14:02.508 / 20250223 23:14:02.508 / 00:00:00.000
00:00:00.000 IF "${status}" == "PASS"  
Start / End / Elapsed: 20250223 23:14:02.508 / 20250223 23:14:02.508 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Fail   ${result}
Documentation:

Fails the test with the given message and optionally alters its tags.
Start / End / Elapsed: 20250223 23:14:02.508 / 20250223 23:14:02.508 / 00:00:00.000
23:14:02.509 INFO Returning from the enclosing user keyword.
23:14:02.509 INFO Returning from the enclosing user keyword.
00:00:00.003 TEARDOWN SSHKeywords . Restore_Current_Ssh_Connection_From_Index   ${current_connection.index}
Documentation:

Restore active SSH connection in SSHLibrary to given index.
Start / End / Elapsed: 20250223 23:14:02.509 / 20250223 23:14:02.512 / 00:00:00.003
23:14:02.513 INFO Returning from the enclosing user keyword.
00:00:00.000 KEYWORD ${listening} = Get Match   ${output} LISTEN
Documentation:

Wrapper around String.Get Regexp Matches to return None if not found or the first match if found.
Start / End / Elapsed: 20250223 23:14:02.513 / 20250223 23:14:02.513 / 00:00:00.000
00:00:00.000 IF '${listening}' == 'None'  
Start / End / Elapsed: 20250223 23:14:02.513 / 20250223 23:14:02.513 / 00:00:00.000
00:00:00.000 KEYWORD BuiltIn . Wait Until Keyword Succeeds   60s 10s ShardStability.Shards_Stability_Get_Details ${DEFAULT_SHARD_LIST} verify_restconf=True
Documentation:

Runs the specified keyword and retries if it fails.
Start / End / Elapsed: 20250223 23:14:02.514 / 20250223 23:14:02.514 / 00:00:00.000
00:00:10.802 SUITE Rpc Provider Precedence
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence
Documentation:

DOMRpcBroker testing: RPC Provider Precedence

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

The aim is to establish that remote RPC implementations have lower priority than local ones, which is to say that any movement of RPCs on remote nodes does not affect routing as long as a local implementation is available.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot
Start / End / Elapsed: 20250223 23:14:02.515 / 20250223 23:14:13.317 / 00:00:10.802
Status: 8 tests total, 8 passed, 0 failed, 0 skipped
00:12:38.032 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 23:14:13.318 / 20250223 23:26:51.350 / 00:12:38.032
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:04:21.316 SUITE Rpc Provider Partition And Heal
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal
Documentation:

DOMRpcBroker testing: RPC Provider Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This tests establishes that the RPC service operates correctly when faced with node failures. This suite supports more than three node cluster setup too.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot
Start / End / Elapsed: 20250223 23:26:51.350 / 20250223 23:31:12.666 / 00:04:21.316
Status: 7 tests total, 2 passed, 5 failed, 0 skipped
00:13:08.023 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 23:31:12.667 / 20250223 23:44:20.690 / 00:13:08.023
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:03:20.272 SUITE Action Provider Precedence
Full Name: controller-clustering-ask.txt.Action Provider Precedence
Documentation:

DOMRpcBroker testing: Action Provider Precedence

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

The aim is to establish that remote action implementations have lower priority than local ones, which is to say that any movement of actions on remote nodes does not affect routing as long as a local implementation is available.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot
Start / End / Elapsed: 20250223 23:44:20.690 / 20250223 23:47:40.962 / 00:03:20.272
Status: 8 tests total, 4 passed, 4 failed, 0 skipped
00:13:00.955 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250223 23:47:40.963 / 20250224 00:00:41.918 / 00:13:00.955
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:04:20.359 SUITE Action Provider Partition And Heal
Full Name: controller-clustering-ask.txt.Action Provider Partition And Heal
Documentation:

DOMRpcBroker testing: RPC Action Provider Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This tests establishes that the RPC service for actions operates correctly when faced with node failures. This suite supports more than three node cluster setup too.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot
Start / End / Elapsed: 20250224 00:00:41.919 / 20250224 00:05:02.278 / 00:04:20.359
Status: 7 tests total, 2 passed, 5 failed, 0 skipped
00:13:09.187 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250224 00:05:02.279 / 20250224 00:18:11.466 / 00:13:09.187
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:02:37.437 SUITE Master Stability
Full Name: controller-clustering-ask.txt.Master Stability
Documentation:

Cluster Singleton testing: Master Stability

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Cluster Singleton service is designed to ensure that only one instance of an application is registered globally in the cluster. The goal is to establish the service operates correctly in face of application registration changing without moving the active instance.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/cluster_singleton/master_stability.robot
Start / End / Elapsed: 20250224 00:18:11.466 / 20250224 00:20:48.903 / 00:02:37.437
Status: 6 tests total, 0 passed, 6 failed, 0 skipped
00:13:08.233 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250224 00:20:48.904 / 20250224 00:33:57.137 / 00:13:08.233
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:02:37.818 SUITE Partition And Heal
Full Name: controller-clustering-ask.txt.Partition And Heal
Documentation:

Cluster Singleton testing: Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Cluster Singleton service is designed to ensure that only one instance of an application is registered globally in the cluster. The goal is to establish the service operates correctly in face of node failures.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot
Start / End / Elapsed: 20250224 00:33:57.137 / 20250224 00:36:34.955 / 00:02:37.818
Status: 6 tests total, 0 passed, 6 failed, 0 skipped
00:13:02.330 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250224 00:36:34.956 / 20250224 00:49:37.286 / 00:13:02.330
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:02:34.915 SUITE Chasing The Leader
Full Name: controller-clustering-ask.txt.Chasing The Leader
Documentation:

Cluster Singleton testing: Chasing the Leader

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test aims to establish the service operates correctly when faced with rapid application transitions without having a stabilized application.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot
Start / End / Elapsed: 20250224 00:49:37.286 / 20250224 00:52:12.201 / 00:02:34.915
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:13:02.316 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250224 00:52:12.202 / 20250224 01:05:14.518 / 00:13:02.316
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:43.404 SUITE Global Rpc Kill
Full Name: controller-clustering-ask.txt.Global Rpc Kill
Documentation:

Controller functional HA testing of global singleton rpcs.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot
Start / End / Elapsed: 20250224 01:05:14.518 / 20250224 01:05:57.922 / 00:00:43.404
Status: 11 tests total, 0 passed, 11 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:00.744 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250224 01:05:57.923 / 20250224 01:18:58.667 / 00:13:00.744
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:43.559 SUITE Global Rpc Freeze
Full Name: controller-clustering-ask.txt.Global Rpc Freeze
Documentation:

Controller functional HA testing of global singleton rpcs if jvm frozen.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot
Start / End / Elapsed: 20250224 01:18:58.667 / 20250224 01:19:42.226 / 00:00:43.559
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:13:03.683 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250224 01:19:42.227 / 20250224 01:32:45.910 / 00:13:03.683
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:43.587 SUITE Global Rpc Isolate
Full Name: controller-clustering-ask.txt.Global Rpc Isolate
Documentation:

Controller functional HA testing of global singleton rpcs.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot
Start / End / Elapsed: 20250224 01:32:45.911 / 20250224 01:33:29.498 / 00:00:43.587
Status: 11 tests total, 0 passed, 11 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:12:58.464 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250224 01:33:29.498 / 20250224 01:46:27.962 / 00:12:58.464
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:33.599 SUITE Carpeople Crud
Full Name: controller-clustering-ask.txt.Carpeople Crud
Documentation:

Suite for performing basic car/people CRUD operations on leaders and followers.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

More precisely, Update operation is not executed, but some operations are using specific RPCs which goes beyond "basic CRUD".

Cars are added by one big PUT to datastore on car Leader. People are added in a loop with add-person RPC on a people Follower. Cars are bought by chunks on each member, by loop with buy-car RPC.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot
Start / End / Elapsed: 20250224 01:46:27.963 / 20250224 01:47:01.562 / 00:00:33.599
Status: 13 tests total, 0 passed, 13 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.170.174:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:32.387 SUITE Car Failover Crud
Full Name: controller-clustering-ask.txt.Car Failover Crud
Documentation:

Suite mixing basic operations with restart of car Leader.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test stops the current leader of the "car" shard and then executes CRD operations on the new leader and a new follower. The stopped member is brought back. This suite uses 3 different car sets, same size but different starting ID.

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot
Start / End / Elapsed: 20250224 01:47:01.563 / 20250224 01:47:33.950 / 00:00:32.387
Status: 16 tests total, 0 passed, 16 failed, 0 skipped
Message: Suite setup failed: HTTPError: 500 Server Error: Internal Server Error for url: http://10.30.170.174:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:33.426 SUITE Car Persistence Recovery
Full Name: controller-clustering-ask.txt.Car Persistence Recovery
Documentation:

This test restarts all controllers to verify recovery of car data from persistence.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot
Start / End / Elapsed: 20250224 01:47:33.951 / 20250224 01:48:07.377 / 00:00:33.426
Status: 7 tests total, 0 passed, 7 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.139:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:32.629 SUITE Buycar Failover
Full Name: controller-clustering-ask.txt.Buycar Failover
Documentation:

This test focuses on testing buy-car RPC over 3 Leader reboots.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

All purchases are against the same node, which is the first one to get rebooted.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot
Start / End / Elapsed: 20250224 01:48:07.377 / 20250224 01:48:40.006 / 00:00:32.629
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.139:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:34.409 SUITE Entity Isolate
Full Name: controller-clustering-ask.txt.Entity Isolate
Documentation:

Suite for performing member isolation and rejoin, we do with entity-ownership leader.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot
Start / End / Elapsed: 20250224 01:48:40.007 / 20250224 01:49:14.416 / 00:00:34.409
Status: 5 tests total, 0 passed, 5 failed, 0 skipped
Message: Suite setup failed: Evaluating expression 'json.loads(\'\'\'{\n "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "error_type": "javax.management.InstanceNotFoundException",\n "request": {\n "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "type": "read"\n },\n "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1073)\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1343)\\n\\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:921)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\\n\\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\\n\\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:233)\\n\\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:245)\\n\\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:221)\\n\\tat org.jolokia.handler.ReadHandler.f... [ Message content over the limit has been removed. ] ...lipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\\n\\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\\n\\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\\n\\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\\n\\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\\n\\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\\n\\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\\n\\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\\n\\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\\n\\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\\n\\tat java.base/java.lang.Thread.run(Thread.java:1583)\\n",\n "status": 404\n}\n\'\'\')' failed: JSONDecodeError: Invalid control character at: line 8 column 190 (char 619)
00:00:32.492 SUITE Buycar Failover Isolation
Full Name: controller-clustering-ask.txt.Buycar Failover Isolation
Documentation:

This test focuses on testing buy-car RPC over the Leader isolation.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

All purchases are against the same node, which is the first one to get isolated.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot
Start / End / Elapsed: 20250224 01:49:14.416 / 20250224 01:49:46.908 / 00:00:32.492
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.139:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:33.275 SUITE Car Failover Crud Isolation
Full Name: controller-clustering-ask.txt.Car Failover Crud Isolation
Documentation:

Suite mixing basic operations with isolation of car Leader.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test isolates the current leader of the "car" shard and then executes CRD operations on the new leader and a new follower. The isolated member is brought back. This suite uses 3 different car sets, same size but different starting ID.

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot
Start / End / Elapsed: 20250224 01:49:46.909 / 20250224 01:50:20.184 / 00:00:33.275
Status: 16 tests total, 0 passed, 16 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.139:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig
00:00:32.575 SUITE Car Outage Corners
Full Name: controller-clustering-ask.txt.Car Outage Corners
Documentation:

Cluster suite for testing minimal and sum-minimal member population behavior.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test stops majority of the followers and verifies car addition is not possible, then resumes single follower (first from original list) and checks that addition works. Then remaining members are brought up. Leader member is always up and assumed to remain Leading during the whole suite run.

TODO: Use initial data to check more operations. TODO: Perhaps merge with car_failover_crud suite.

Other modules and Shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-titanium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot
Start / End / Elapsed: 20250224 01:50:20.184 / 20250224 01:50:52.759 / 00:00:32.575
Status: 8 tests total, 0 passed, 8 failed, 0 skipped
Message: Suite setup failed: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.139:8181/rests/data/ietf-yang-library:modules-state?content=nonconfig