Test Statistics

Total Statistics
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
All Tests
19521174003:31:12
Statistics by Tag
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
carpeople
85085000:00:00
clustering
85085000:00:00
critical
19521174003:31:12
Statistics by Suite
Total
Pass
Fail
Skip
Elapsed
Pass / Fail / Skip
controller-clustering-ask.txt
19521174003:57:23
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:12:32
controller-clustering-ask . txt . Rpc Provider Precedence
817000:06:18
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:25
controller-clustering-ask . txt . Rpc Provider Partition And Heal
725000:14:00
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:13
controller-clustering-ask . txt . Action Provider Precedence
844000:14:41
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:23
controller-clustering-ask . txt . Action Provider Partition And Heal
725000:14:00
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:25
controller-clustering-ask . txt . Master Stability
606000:09:08
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:13:19
controller-clustering-ask . txt . Partition And Heal
606000:05:06
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:26
controller-clustering-ask . txt . Chasing The Leader
312000:08:07
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:15
controller-clustering-ask . txt . Global Rpc Kill
11011000:01:04
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:24
controller-clustering-ask . txt . Global Rpc Freeze
10010000:01:05
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:23
controller-clustering-ask . txt . Global Rpc Isolate
11011000:01:05
controller-clustering-ask . txt . Restart Odl With Tell Based False
312000:14:21
controller-clustering-ask . txt . Carpeople Crud
13013000:01:01
controller-clustering-ask . txt . Car Failover Crud
16016000:00:59
controller-clustering-ask . txt . Car Persistence Recovery
707000:00:58
controller-clustering-ask . txt . Buycar Failover
10010000:00:59
controller-clustering-ask . txt . Entity Isolate
505000:00:54
controller-clustering-ask . txt . Buycar Failover Isolation
10010000:00:58
controller-clustering-ask . txt . Car Failover Crud Isolation
16016000:00:58
controller-clustering-ask . txt . Car Outage Corners
808000:00:59

Test Execution Log

03:57:23.415 SUITE controller-clustering-ask.txt
Full Name: controller-clustering-ask.txt
Start / End / Elapsed: 20250323 23:08:52.037 / 20250324 03:06:15.452 / 03:57:23.415
Status: 195 tests total, 21 passed, 174 failed, 0 skipped
00:12:31.679 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250323 23:08:52.059 / 20250323 23:21:23.738 / 00:12:31.679
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:00:04.453 SETUP SetupUtils . Setup_Utils_For_Setup_And_Teardown   http_timeout=125
Documentation:

Prepare both FailFast and Karaf logging, log that the caller suite has started. This keyword is to be used in suite setup, and without it the other keywords may not work properly.
Start / End / Elapsed: 20250323 23:08:52.600 / 20250323 23:08:57.053 / 00:00:04.453
00:00:00.000 TEARDOWN SSHLibrary . Close All Connections  
Documentation:

Closes all open connections.
Start / End / Elapsed: 20250323 23:21:23.737 / 20250323 23:21:23.737 / 00:00:00.000
00:06:12.530 TEST Stop_All_Members
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Stop_All_Members
Documentation:

Stop every odl node.
Tags: critical
Start / End / Elapsed: 20250323 23:08:57.053 / 20250323 23:15:09.583 / 00:06:12.530
Status: FAIL
Message: Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1
00:00:09.534 TEST Unset_Tell_Based_Protocol_Usage
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Unset_Tell_Based_Protocol_Usage
Documentation:

Comment out the flag usage in config file. Also clean most data except data/log/.
Tags: critical
Start / End / Elapsed: 20250323 23:15:09.583 / 20250323 23:15:19.117 / 00:00:09.534
Status: PASS
00:06:04.619 TEST Start_All_And_Sync
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False.Start_All_And_Sync
Documentation:

Start each member and wait for sync.
Tags: critical
Start / End / Elapsed: 20250323 23:15:19.117 / 20250323 23:21:23.736 / 00:06:04.619
Status: FAIL
Message: Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.39:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore
00:06:18.184 SUITE Rpc Provider Precedence
Full Name: controller-clustering-ask.txt.Rpc Provider Precedence
Documentation:

DOMRpcBroker testing: RPC Provider Precedence

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

The aim is to establish that remote RPC implementations have lower priority than local ones, which is to say that any movement of RPCs on remote nodes does not affect routing as long as a local implementation is available.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot
Start / End / Elapsed: 20250323 23:21:23.738 / 20250323 23:27:41.922 / 00:06:18.184
Status: 8 tests total, 1 passed, 7 failed, 0 skipped
00:14:24.803 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250323 23:27:41.923 / 20250323 23:42:06.726 / 00:14:24.803
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:14:00.323 SUITE Rpc Provider Partition And Heal
Full Name: controller-clustering-ask.txt.Rpc Provider Partition And Heal
Documentation:

DOMRpcBroker testing: RPC Provider Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This tests establishes that the RPC service operates correctly when faced with node failures. This suite supports more than three node cluster setup too.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot
Start / End / Elapsed: 20250323 23:42:06.726 / 20250323 23:56:07.049 / 00:14:00.323
Status: 7 tests total, 2 passed, 5 failed, 0 skipped
00:14:12.983 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250323 23:56:07.050 / 20250324 00:10:20.033 / 00:14:12.983
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:14:41.056 SUITE Action Provider Precedence
Full Name: controller-clustering-ask.txt.Action Provider Precedence
Documentation:

DOMRpcBroker testing: Action Provider Precedence

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

The aim is to establish that remote action implementations have lower priority than local ones, which is to say that any movement of actions on remote nodes does not affect routing as long as a local implementation is available.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot
Start / End / Elapsed: 20250324 00:10:20.034 / 20250324 00:25:01.090 / 00:14:41.056
Status: 8 tests total, 4 passed, 4 failed, 0 skipped
00:14:22.698 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250324 00:25:01.090 / 20250324 00:39:23.788 / 00:14:22.698
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:14:00.236 SUITE Action Provider Partition And Heal
Full Name: controller-clustering-ask.txt.Action Provider Partition And Heal
Documentation:

DOMRpcBroker testing: RPC Action Provider Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This tests establishes that the RPC service for actions operates correctly when faced with node failures. This suite supports more than three node cluster setup too.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot
Start / End / Elapsed: 20250324 00:39:23.788 / 20250324 00:53:24.024 / 00:14:00.236
Status: 7 tests total, 2 passed, 5 failed, 0 skipped
00:14:24.766 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250324 00:53:24.024 / 20250324 01:07:48.790 / 00:14:24.766
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:09:07.926 SUITE Master Stability
Full Name: controller-clustering-ask.txt.Master Stability
Documentation:

Cluster Singleton testing: Master Stability

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Cluster Singleton service is designed to ensure that only one instance of an application is registered globally in the cluster. The goal is to establish the service operates correctly in face of application registration changing without moving the active instance.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/cluster_singleton/master_stability.robot
Start / End / Elapsed: 20250324 01:07:48.791 / 20250324 01:16:56.717 / 00:09:07.926
Status: 6 tests total, 0 passed, 6 failed, 0 skipped
00:13:19.186 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250324 01:16:56.718 / 20250324 01:30:15.904 / 00:13:19.186
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:05:06.152 SUITE Partition And Heal
Full Name: controller-clustering-ask.txt.Partition And Heal
Documentation:

Cluster Singleton testing: Partition And Heal

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Cluster Singleton service is designed to ensure that only one instance of an application is registered globally in the cluster. The goal is to establish the service operates correctly in face of node failures.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot
Start / End / Elapsed: 20250324 01:30:15.905 / 20250324 01:35:22.057 / 00:05:06.152
Status: 6 tests total, 0 passed, 6 failed, 0 skipped
00:14:25.528 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250324 01:35:22.057 / 20250324 01:49:47.585 / 00:14:25.528
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:08:06.603 SUITE Chasing The Leader
Full Name: controller-clustering-ask.txt.Chasing The Leader
Documentation:

Cluster Singleton testing: Chasing the Leader

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test aims to establish the service operates correctly when faced with rapid application transitions without having a stabilized application.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot
Start / End / Elapsed: 20250324 01:49:47.586 / 20250324 01:57:54.189 / 00:08:06.603
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:14:14.776 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250324 01:57:54.189 / 20250324 02:12:08.965 / 00:14:14.776
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:01:04.152 SUITE Global Rpc Kill
Full Name: controller-clustering-ask.txt.Global Rpc Kill
Documentation:

Controller functional HA testing of global singleton rpcs.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot
Start / End / Elapsed: 20250324 02:12:08.966 / 20250324 02:13:13.118 / 00:01:04.152
Status: 11 tests total, 0 passed, 11 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:14:23.622 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250324 02:13:13.119 / 20250324 02:27:36.741 / 00:14:23.622
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:01:05.350 SUITE Global Rpc Freeze
Full Name: controller-clustering-ask.txt.Global Rpc Freeze
Documentation:

Controller functional HA testing of global singleton rpcs if jvm frozen.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot
Start / End / Elapsed: 20250324 02:27:36.741 / 20250324 02:28:42.091 / 00:01:05.350
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:14:23.334 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250324 02:28:42.092 / 20250324 02:43:05.426 / 00:14:23.334
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:01:04.726 SUITE Global Rpc Isolate
Full Name: controller-clustering-ask.txt.Global Rpc Isolate
Documentation:

Controller functional HA testing of global singleton rpcs.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot
Start / End / Elapsed: 20250324 02:43:05.427 / 20250324 02:44:10.153 / 00:01:04.726
Status: 11 tests total, 0 passed, 11 failed, 0 skipped
Message: Suite setup failed: No match found for 'opendaylight-user' in 10 seconds Output: .
00:14:20.596 SUITE Restart Odl With Tell Based False
Full Name: controller-clustering-ask.txt.Restart Odl With Tell Based False
Documentation:

Unset tell-based protocol usage

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Suite stops all odl nodes, outcomment usage of tell-based protocol in config file (means make it false by default) and starts all nodes again.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot
Start / End / Elapsed: 20250324 02:44:10.154 / 20250324 02:58:30.750 / 00:14:20.596
Status: 3 tests total, 1 passed, 2 failed, 0 skipped
00:01:00.608 SUITE Carpeople Crud
Full Name: controller-clustering-ask.txt.Carpeople Crud
Documentation:

Suite for performing basic car/people CRUD operations on leaders and followers.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

More precisely, Update operation is not executed, but some operations are using specific RPCs which goes beyond "basic CRUD".

Cars are added by one big PUT to datastore on car Leader. People are added in a loop with add-person RPC on a people Follower. Cars are bought by chunks on each member, by loop with buy-car RPC.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot
Start / End / Elapsed: 20250324 02:58:30.751 / 20250324 02:59:31.359 / 00:01:00.608
Status: 13 tests total, 0 passed, 13 failed, 0 skipped
Message: Suite setup failed: ReadTimeout: HTTPConnectionPool(host='10.30.171.58', port=8181): Read timed out. (read timeout=5.0)
00:00:58.937 SUITE Car Failover Crud
Full Name: controller-clustering-ask.txt.Car Failover Crud
Documentation:

Suite mixing basic operations with restart of car Leader.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test stops the current leader of the "car" shard and then executes CRD operations on the new leader and a new follower. The stopped member is brought back. This suite uses 3 different car sets, same size but different starting ID.

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot
Start / End / Elapsed: 20250324 02:59:31.359 / 20250324 03:00:30.296 / 00:00:58.937
Status: 16 tests total, 0 passed, 16 failed, 0 skipped
Message: Suite setup failed: ReadTimeout: HTTPConnectionPool(host='10.30.171.58', port=8181): Read timed out. (read timeout=5.0)
00:00:57.691 SUITE Car Persistence Recovery
Full Name: controller-clustering-ask.txt.Car Persistence Recovery
Documentation:

This test restarts all controllers to verify recovery of car data from persistence.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot
Start / End / Elapsed: 20250324 03:00:30.297 / 20250324 03:01:27.988 / 00:00:57.691
Status: 7 tests total, 0 passed, 7 failed, 0 skipped
Message: Suite setup failed: ReadTimeout: HTTPConnectionPool(host='10.30.171.58', port=8181): Read timed out. (read timeout=5.0)
00:00:58.608 SUITE Buycar Failover
Full Name: controller-clustering-ask.txt.Buycar Failover
Documentation:

This test focuses on testing buy-car RPC over 3 Leader reboots.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

All purchases are against the same node, which is the first one to get rebooted.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot
Start / End / Elapsed: 20250324 03:01:27.989 / 20250324 03:02:26.597 / 00:00:58.608
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: ReadTimeout: HTTPConnectionPool(host='10.30.171.58', port=8181): Read timed out. (read timeout=5.0)
00:00:54.324 SUITE Entity Isolate
Full Name: controller-clustering-ask.txt.Entity Isolate
Documentation:

Suite for performing member isolation and rejoin, we do with entity-ownership leader.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot
Start / End / Elapsed: 20250324 03:02:26.597 / 20250324 03:03:20.921 / 00:00:54.324
Status: 5 tests total, 0 passed, 5 failed, 0 skipped
Message: Suite setup failed: Evaluating expression 'json.loads(\'\'\'{\n "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "error_type": "javax.management.InstanceNotFoundException",\n "request": {\n "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "type": "read"\n },\n "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1073)\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1343)\\n\\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:921)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\\n\\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\\n\\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:233)\\n\\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:245)\\n\\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:221)\\n\\tat org.jolokia.handler.ReadHandler.f... [ Message content over the limit has been removed. ] ...lipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\\n\\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\\n\\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\\n\\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\\n\\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\\n\\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\\n\\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\\n\\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\\n\\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\\n\\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\\n\\tat java.base/java.lang.Thread.run(Thread.java:1583)\\n",\n "status": 404\n}\n\'\'\')' failed: JSONDecodeError: Invalid control character at: line 8 column 190 (char 619)
00:00:57.812 SUITE Buycar Failover Isolation
Full Name: controller-clustering-ask.txt.Buycar Failover Isolation
Documentation:

This test focuses on testing buy-car RPC over the Leader isolation.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

All purchases are against the same node, which is the first one to get isolated.

All data is deleted at the end of the suite. This suite expects car, people and car-people modules to have separate Shards.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot
Start / End / Elapsed: 20250324 03:03:20.922 / 20250324 03:04:18.734 / 00:00:57.812
Status: 10 tests total, 0 passed, 10 failed, 0 skipped
Message: Suite setup failed: ReadTimeout: HTTPConnectionPool(host='10.30.171.58', port=8181): Read timed out. (read timeout=5.0)
00:00:57.933 SUITE Car Failover Crud Isolation
Full Name: controller-clustering-ask.txt.Car Failover Crud Isolation
Documentation:

Suite mixing basic operations with isolation of car Leader.

Copyright (c) 2017 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test isolates the current leader of the "car" shard and then executes CRD operations on the new leader and a new follower. The isolated member is brought back. This suite uses 3 different car sets, same size but different starting ID.

Other models and shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot
Start / End / Elapsed: 20250324 03:04:18.734 / 20250324 03:05:16.667 / 00:00:57.933
Status: 16 tests total, 0 passed, 16 failed, 0 skipped
Message: Suite setup failed: ReadTimeout: HTTPConnectionPool(host='10.30.171.58', port=8181): Read timed out. (read timeout=5.0)
00:00:58.783 SUITE Car Outage Corners
Full Name: controller-clustering-ask.txt.Car Outage Corners
Documentation:

Cluster suite for testing minimal and sum-minimal member population behavior.

Copyright (c) 2016 Cisco Systems, Inc. and others. All rights reserved.

This program and the accompanying materials are made available under the terms of the Eclipse Public License v1.0 which accompanies this distribution, and is available at http://www.eclipse.org/legal/epl-v10.html

This test stops majority of the followers and verifies car addition is not possible, then resumes single follower (first from original list) and checks that addition works. Then remaining members are brought up. Leader member is always up and assumed to remain Leading during the whole suite run.

TODO: Use initial data to check more operations. TODO: Perhaps merge with car_failover_crud suite.

Other modules and Shards (people, car-people) are not accessed by this suite.

All data is deleted at the end of the suite. This suite expects car module to have a separate Shard.
Source: /w/workspace/controller-csit-3node-clustering-ask-all-scandium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot
Start / End / Elapsed: 20250324 03:05:16.667 / 20250324 03:06:15.450 / 00:00:58.783
Status: 8 tests total, 0 passed, 8 failed, 0 skipped
Message: Suite setup failed: ReadTimeout: HTTPConnectionPool(host='10.30.171.58', port=8181): Read timed out. (read timeout=5.0)