12:23:37,338 PM [test-akka.actor.default-dispatcher-4] [INFO] Slf4jLogger - Slf4jLogger started 12:23:37,359 PM [test-akka.actor.default-dispatcher-4] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://test@127.0.0.1:2565] with UID [4363875674549386964] 12:23:37,360 PM [test-akka.actor.default-dispatcher-4] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Starting up, Akka version [2.6.21] ... 12:23:37,369 PM [test-akka.actor.default-dispatcher-4] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Registered cluster JMX MBean [akka:type=Cluster] 12:23:37,369 PM [test-akka.actor.default-dispatcher-4] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Started up successfully 12:23:37,369 PM [test-akka.actor.default-dispatcher-4] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:37,369 PM [test-akka.actor.default-dispatcher-4] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:37,546 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$a#-258666667] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:37,559 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config1 12:23:37,621 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config1 12:23:37,621 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Creating local shard: member-1-shard-default-config1 12:23:37,638 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:37,638 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config1 12:23:37,638 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Initializing shard [default] 12:23:37,638 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config1 is initialized 12:23:37,638 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config1: Received role changed for member-1-shard-default-config1 from null to Follower 12:23:37,639 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: onChangeShardServersVotingStatus: ChangeShardMembersVotingStatus [shardName=default, meberVotingStatusMap={member-2=true}] 12:23:37,640 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: findLocalShard : default 12:23:37,643 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Sending ChangeServersVotingStatus message ChangeServersVotingStatus [serverVotingStatusMap={member-2-shard-default-config1=true}, serversVisited=[]] to local shard akka://test/user/member-1-shard-default-config1 12:23:37,644 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Received ServerChangeReply{status=NO_LEADER} from local shard akka://test/user/member-1-shard-default-config1 12:23:37,644 PM [test-akka.actor.default-dispatcher-10] [WARN] ShardManager - shard-manager-config1: ChangeServersVotingStatus failed for shard default with status NO_LEADER 12:23:37,807 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config1#614827686] 12:23:37,822 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForShardLeader starting 12:23:37,825 PM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$b#-510637540] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:37,826 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config2 12:23:37,827 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Recovery complete : shard-manager-config2 12:23:37,827 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config2: Creating local shard: member-1-shard-default-config2 12:23:37,837 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:37,837 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config2 12:23:37,841 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config2: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:37,848 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config2: Scheduling 10000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config2, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 12:23:37,987 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config2: Initializing shard [default] 12:23:37,987 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized 12:23:37,987 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized - notifying 1 OnShardInitialized callbacks 12:23:38,138 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config2: Received role changed for member-1-shard-default-config2 from Candidate to Leader 12:23:38,138 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized - notifying 1 OnShardInitialized callbacks 12:23:38,289 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config2: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config2, leaderId=member-1-shard-default-config2, leaderPayloadVersion=13] 12:23:38,289 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is ready - notifying 1 OnShardInitialized callbacks 12:23:38,289 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config2: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#1442644644, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$fcdTIJnO@ce46635] 12:23:38,290 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config2: All Shards are ready - data store config2 is ready 12:23:38,490 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForShardLeader ending 12:23:38,591 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$b#-510637540] 12:23:38,604 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config2 12:23:38,615 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/$$a] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:38,615 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config3 12:23:38,617 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config3 12:23:38,617 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config3: Creating local shard: member-1-shard-default-config3 12:23:38,618 PM [main] [INFO] ShardManager - shard-manager-config3: Received role changed for member-1-shard-default-config3 from Follower to Candidate 12:23:38,618 PM [main] [INFO] ShardManager - shard-manager-config3 Received follower initial sync status for member-1-shard-default-config3 status sync done true 12:23:38,719 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$a] 12:23:38,730 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config3 12:23:38,741 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 12:23:38,751 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [2475980477050432010] 12:23:38,751 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 12:23:38,758 PM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:23:38,758 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 12:23:38,763 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:38,763 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:38,779 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config4 12:23:38,781 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config4] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:38,790 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 12:23:38,792 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 12:23:38,800 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 12:23:38,820 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 12:23:38,833 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config4 12:23:38,833 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-1-shard-default-config4 12:23:38,833 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-1-shard-astronauts-config4 12:23:38,833 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 12:23:38,834 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 12:23:38,834 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 12:23:38,867 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-8899224341433562076] 12:23:38,868 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 12:23:38,870 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:23:38,870 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 12:23:38,871 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:38,871 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:38,882 PM [main] [ERROR] ShardManagerTest - Mock Shard Leader Actor : TestActor[akka://cluster-test/user/member-2-shard-default-config4] 12:23:38,902 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config4] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:38,913 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config4 12:23:38,914 PM [main] [ERROR] ShardManagerTest - Forwarding actor : Actor[akka://cluster-test/user/shardmanager-config4/member-2-shard-default-config4#-2110506761] 12:23:38,924 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:38,924 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config4 12:23:38,924 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 12:23:38,924 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 to akka://cluster-test/user/member-1-shard-default-config 12:23:38,925 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config4: updated peer member-1-shard-default-config4 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 on shard member-1-shard-default-config4 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#957550527] 12:23:38,925 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-astronauts-config4 12:23:38,925 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 12:23:38,925 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 to akka://cluster-test/user/member-1-shard-default-config 12:23:38,925 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config4: updated peer member-1-shard-astronauts-config4 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 on shard member-1-shard-astronauts-config4 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#957550527] 12:23:38,925 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config4: Initializing shard [default] 12:23:38,925 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config4 is initialized 12:23:38,925 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config4: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config4, leaderId=member-2-shard-default-config4, leaderPayloadVersion=12] 12:23:38,925 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config4: Received role changed for member-1-shard-default-config4 from Candidate to Follower 12:23:38,958 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config4 12:23:38,958 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-2-shard-default-config4 12:23:38,959 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:38,959 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config4 12:23:38,959 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config4: Initializing shard [default] 12:23:38,959 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-2-shard-default-config4 is initialized 12:23:38,959 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config4: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config4, leaderId=member-2-shard-default-config4, leaderPayloadVersion=12] 12:23:38,959 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config4: Received role changed for member-2-shard-default-config4 from Candidate to Leader 12:23:38,959 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 12:23:39,277 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 12:23:39,348 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 12:23:39,349 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 12:23:39,349 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 12:23:39,349 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 to akka://cluster-test/user/member-2-shard-default-config4 12:23:39,349 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 12:23:39,789 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 12:23:39,789 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:23:39,789 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 12:23:39,789 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 to akka://cluster-test/user/member-1-shard-default-config 12:23:39,789 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-astronauts-config4 12:23:39,789 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-astronauts-config4 to akka://cluster-test/user/member-1-shard-default-config 12:23:39,801 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: onRemoveShardReplica: RemoveShardReplica [shardName=default, memberName=MemberName{name=member-1}] 12:23:39,806 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:39,806 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4, primaryVersion=12] 12:23:39,807 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: Sending RemoveServer message to peer akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 for shard member-1-shard-default-config4 12:23:39,829 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config4: Received RemoveServerReply{status=OK, leaderHint=member-2-shard-default-config4} for shard default 12:23:39,829 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config4: Leader shard successfully removed the replica shard default 12:23:39,900 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:23:39,900 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 12:23:39,900 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 to akka://cluster-test/user/member-2-shard-default-config4 12:23:39,900 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 12:23:39,943 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config4 12:23:39,947 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:23:39,947 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:23:39,975 PM [cluster-test-akka.actor.default-dispatcher-14] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:23:39,976 PM [cluster-test-akka.actor.default-dispatcher-14] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:23:39,983 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2558], message stream] Upstream failed, cause: StreamTcpException: The connection closed with error: Connection reset 12:23:39,983 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2558], control stream] Upstream failed, cause: StreamTcpException: The connection closed with error: Connection reset 12:23:40,28 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config4 12:23:40,29 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:23:40,29 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:23:41,48 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 12:23:41.050] [cluster-test-akka.actor.internal-dispatcher-4] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:23:41,63 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$c#1780711257] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:41,63 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config5 12:23:41,64 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Recovery complete : shard-manager-config5 12:23:41,64 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config5: Creating local shard: member-1-shard-default-config5 12:23:41,74 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:41,74 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config5 12:23:41,74 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config5: Initializing shard [default] 12:23:41,74 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config5 is initialized 12:23:41,226 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$c#1780711257] 12:23:41,237 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config5 12:23:41,240 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config6 12:23:41,240 PM [test-akka.actor.default-dispatcher-4] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/$$b] to Actor[akka://test/user/termination-monitor] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:41,243 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Recovery complete : shard-manager-config6 12:23:41,243 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config6: Creating local shard: member-1-shard-default-config6 12:23:41,244 PM [main] [INFO] ShardManager - shard-manager-config6: Received role changed for member-1-shard-default-config6 from null to Follower 12:23:41,254 PM [main] [INFO] ShardManager - shard-manager-config6: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 12:23:41,254 PM [main] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config6 with address akka://test/user/shardmanager-config6/member-2-shard-default-config6 12:23:41,254 PM [main] [INFO] ShardManager - shard-manager-config6: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config6, leaderId=member-2-shard-default-config6, leaderPayloadVersion=13] 12:23:41,254 PM [main] [INFO] ShardManager - shard-manager-config6: All Shards are ready - data store config6 is ready 12:23:41,355 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$b] 12:23:41,366 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config6 12:23:41,372 PM [test-akka.actor.default-dispatcher-4] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$d#151371284] to Actor[akka://test/user/termination-monitor] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:41,373 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config7 12:23:41,375 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config7 12:23:41,383 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config7: onRemoveShardReplica: RemoveShardReplica [shardName=model-inventory, memberName=MemberName{name=member-1}] 12:23:41,383 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config7: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=model-inventory, waitUntilReady=true] 12:23:41,383 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config7: No shard found for model-inventory 12:23:41,385 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config7: Failed to find leader for shard model-inventory: received response: {} org.opendaylight.controller.cluster.datastore.exceptions.PrimaryNotFoundException: No primary shard found for model-inventory. at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.findPrimary(ShardManager.java:1051) at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.handleCommand(ShardManager.java:207) at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManagerTest$TestShardManager.handleCommand(ShardManagerTest.java:2199) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) at scala.PartialFunction.applyOrElse(PartialFunction.scala:214) at scala.PartialFunction.applyOrElse$(PartialFunction.scala:213) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:537) at akka.actor.Actor.aroundReceive$(Actor.scala:535) at akka.persistence.AbstractPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:295) at akka.persistence.Eventsourced$$anon$4.stateReceive(Eventsourced.scala:917) at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:245) at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:244) at akka.persistence.AbstractPersistentActor.aroundReceive(PersistentActor.scala:295) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579) at akka.actor.ActorCell.invoke(ActorCell.scala:547) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) at akka.dispatch.Mailbox.run(Mailbox.scala:231) at akka.dispatch.Mailbox.exec(Mailbox.scala:243) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) 12:23:41,487 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$d#151371284] 12:23:41,498 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config7 12:23:41,500 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingLocalReplicaLeader starting 12:23:41,508 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$e#238424325] to Actor[akka://test/user/termination-monitor] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:41,509 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config8 12:23:41,510 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Recovery complete : shard-manager-config8 12:23:41,510 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config8: Creating local shard: member-1-shard-default-config8 12:23:41,520 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:41,520 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config8 12:23:41,520 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config8: Initializing shard [default] 12:23:41,520 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config8 is initialized 12:23:41,520 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config8: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config8, leaderId=member-1-shard-default-config8, leaderPayloadVersion=13] 12:23:41,520 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config8: Received role changed for member-1-shard-default-config8 from Candidate to Leader 12:23:41,520 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config8: All Shards are ready - data store config8 is ready 12:23:41,520 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config8: onAddShardReplica: AddShardReplica[ShardName=default] 12:23:41,521 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config8: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:41,521 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config8: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#33845629, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$fcdTIJnO@5070d54c] 12:23:41,521 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config8: Local shard default already exists 12:23:41,522 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config8: findLocalShard : default 12:23:41,522 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingLocalReplicaLeader ending 12:23:41,623 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$e#238424325] 12:23:41,634 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Stopping ShardManager shard-manager-config8 12:23:41,636 PM [main] [INFO] ShardManagerTest - testOnReceiveFindLocalShardWaitForShardInitialized starting 12:23:41,639 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$f#-1455629918] to Actor[akka://test/user/termination-monitor] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:41,639 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Starting ShardManager shard-manager-config9 12:23:41,641 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config9 12:23:41,641 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config9: Creating local shard: member-1-shard-default-config9 12:23:41,650 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:41,650 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config9 12:23:41,650 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config9: findLocalShard : default 12:23:41,650 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config9: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config9, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 12:23:41,650 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config9: Initializing shard [default] 12:23:41,650 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-default-config9 is initialized 12:23:41,650 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-default-config9 is initialized - notifying 1 OnShardInitialized callbacks 12:23:41,656 PM [main] [INFO] ShardManagerTest - testOnReceiveFindLocalShardWaitForShardInitialized starting 12:23:41,757 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$f#-1455629918] 12:23:41,768 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config9 12:23:41,770 PM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberUnreachableAndLeadershipChange starting 12:23:41,781 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 12:23:41,792 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [-2255443714816211336] 12:23:41,797 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 12:23:41,797 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 12:23:41,797 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 12:23:41,797 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:41,797 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:41,812 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 12:23:41,812 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 12:23:41,812 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 12:23:41,813 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 12:23:41,817 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config10] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:41,817 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config10 12:23:41,832 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-2082295151254435860] 12:23:41,832 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 12:23:41,832 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:23:41,832 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 12:23:41,856 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config10 12:23:41,857 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config10: Creating local shard: member-1-shard-default-config10 12:23:41,857 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 12:23:41,857 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 12:23:41,869 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config10 12:23:41,876 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:41,876 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config10 12:23:41,876 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 12:23:41,876 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 to akka://cluster-test/user/member-1-shard-default-config 12:23:41,876 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config10: updated peer member-1-shard-default-config10 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 on shard member-1-shard-default-config10 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#-610148591] 12:23:41,876 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config10: Initializing shard [default] 12:23:41,876 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config10 is initialized 12:23:41,876 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config10, leaderId=member-2-shard-default-config10, leaderPayloadVersion=13] 12:23:41,876 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config10: Received role changed for member-1-shard-default-config10 from Candidate to Follower 12:23:41,883 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config10 12:23:41,883 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config10: Creating local shard: member-2-shard-default-config10 12:23:41,888 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:41,888 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:41,888 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config10] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:41,895 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:41,895 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config10 12:23:41,895 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config10: Initializing shard [default] 12:23:41,895 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-2-shard-default-config10 is initialized 12:23:41,895 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config10, leaderId=member-2-shard-default-config10, leaderPayloadVersion=13] 12:23:41,895 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config10: Received role changed for member-2-shard-default-config10 from Candidate to Leader 12:23:41,895 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 12:23:41,986 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 12:23:41,998 PM [cluster-test-akka.actor.default-dispatcher-18] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 12:23:41,999 PM [cluster-test-akka.actor.default-dispatcher-18] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 12:23:41,999 PM [cluster-test-akka.actor.default-dispatcher-18] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 12:23:41,999 PM [cluster-test-akka.actor.default-dispatcher-18] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 to akka://cluster-test/user/member-2-shard-default-config 12:23:41,999 PM [cluster-test-akka.actor.default-dispatcher-18] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 12:23:42,809 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 12:23:42,812 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:23:42,812 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 12:23:42,812 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 to akka://cluster-test/user/member-1-shard-default-config 12:23:42,812 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 12:23:42,820 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:42,820 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10, primaryVersion=13] 12:23:42,821 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 12:23:42,821 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config10 as unavailable. 12:23:42,822 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:42,822 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config10, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config10, activeMember=true] 12:23:42,858 PM [cluster-test-akka.actor.default-dispatcher-19] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:23:42,858 PM [cluster-test-akka.actor.default-dispatcher-19] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 12:23:42,858 PM [cluster-test-akka.actor.default-dispatcher-19] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 to akka://cluster-test/user/member-2-shard-default-config 12:23:42,858 PM [cluster-test-akka.actor.default-dispatcher-19] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 12:23:44,36 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Received ShardNotInitializedTimeout message for shard default 12:23:44,36 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Returning NoShardLeaderException for shard default 12:23:44,37 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config10, leaderId=member-1-shard-default-config10, leaderPayloadVersion=13] 12:23:44,37 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 12:23:44,37 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received role changed for member-1-shard-default-config10 from Follower to Leader 12:23:44,37 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 12:23:44,37 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:44,37 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/member-1-shard-default-config#-610148591, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$fcdTIJnO@61d70a1e] 12:23:44,37 PM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberUnreachableAndLeadershipChange ending 12:23:44,139 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config10 12:23:44,140 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:23:44,140 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:23:44,162 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:23:44,162 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:23:44,165 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 12:23:44,178 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config10 12:23:44,180 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:23:44,180 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:23:45,193 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 12:23:45.195] [cluster-test-akka.actor.internal-dispatcher-2] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:23:45,226 PM [test-akka.actor.default-dispatcher-4] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/shardmanager-config11] to Actor[akka://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:45,227 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config11 12:23:45,229 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config11 12:23:45,249 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:45,249 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config11: onAddShardReplica: AddShardReplica[ShardName=astronauts] 12:23:45,252 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config11: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-530#-976692957 for shard member-1-shard-astronauts-config11 12:23:45,263 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config11: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-2}] 12:23:45,263 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config11: A shard replica operation for astronauts is already in progress 12:23:45,367 PM [main] [INFO] ShardManagerTest - testPerShardDatastoreContext starting 12:23:45,375 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Starting ShardManager shard-manager-config12 12:23:45,376 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config12 12:23:45,376 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config12: Creating local shard: member-1-shard-default-config12 12:23:45,376 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config12: Creating local shard: member-1-shard-topology-config12 12:23:45,378 PM [test-akka.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$g#-1171427254] to Actor[akka://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:45,385 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:45,385 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config12 12:23:45,385 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-topology-config12 12:23:45,388 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-default-config12 12:23:45,388 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-topology-config12 12:23:45,438 PM [main] [INFO] ShardManagerTest - testPerShardDatastoreContext ending 12:23:45,540 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/default1#1607790590] 12:23:45,550 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/topology2#1349145112] 12:23:45,551 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$g#-1171427254] 12:23:45,551 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config12 12:23:45,554 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForLocalLeaderShard starting 12:23:45,561 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config13 12:23:45,564 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config13 12:23:45,564 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config13: Creating local shard: member-1-shard-default-config13 12:23:45,572 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:45,572 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config13 12:23:45,572 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config13: Initializing shard [default] 12:23:45,572 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-default-config13 is initialized 12:23:45,572 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - shard-manager-config13: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config13, leaderId=member-1-shard-default-config13, leaderPayloadVersion=13] 12:23:45,572 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - shard-manager-config13: Received role changed for member-1-shard-default-config13 from Candidate to Leader 12:23:45,572 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - shard-manager-config13: All Shards are ready - data store config13 is ready 12:23:45,572 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config13: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 12:23:45,572 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config13: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#918340247, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$fcdTIJnO@31d3b794] 12:23:45,572 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForLocalLeaderShard ending 12:23:45,674 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$h#145539213] 12:23:45,685 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Stopping ShardManager shard-manager-config13 12:23:45,687 PM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberWithNameContainedInLeaderIdUnreachable starting 12:23:45,699 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 12:23:45,710 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2562] with UID [-6947803197609617774] 12:23:45,710 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Starting up, Akka version [2.6.21] ... 12:23:45,712 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Registered cluster JMX MBean [akka:type=Cluster] 12:23:45,712 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Started up successfully 12:23:45,729 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:45,729 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:45,731 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Node [akka://cluster-test@127.0.0.1:2562] is JOINING itself (with roles [member-256, dc-default], version [0.0.0]) and forming new cluster 12:23:45,731 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - is the new leader among reachable nodes (more leaders may exist) 12:23:45,731 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Leader is moving node [akka://cluster-test@127.0.0.1:2562] to [Up] 12:23:45,731 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config14] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:45,740 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config14 12:23:45,745 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config14 12:23:45,746 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config14: Creating local shard: member-256-shard-default-config14 12:23:45,746 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-256}, address: akka://cluster-test@127.0.0.1:2562 12:23:45,746 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 12:23:45,753 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 12:23:45,770 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-8652731577149595684] 12:23:45,770 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 12:23:45,772 PM [cluster-test-akka.actor.default-dispatcher-6] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:23:45,772 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 12:23:45,780 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:45,780 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:45,789 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config14] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:45,791 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config14 12:23:45,796 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Recovery complete : shard-manager-config14 12:23:45,796 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config14: Creating local shard: member-2-shard-default-config14 12:23:45,796 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:45,796 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config14 12:23:45,796 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config14: Initializing shard [default] 12:23:45,796 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-2-shard-default-config14 is initialized 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config14, leaderId=member-256-shard-default-config14, leaderPayloadVersion=13] 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-256-shard-default-config14 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: Received role changed for member-2-shard-default-config14 from Candidate to Follower 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 to akka://cluster-test/user/member-256-shard-default-config 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config14: updated peer member-256-shard-default-config14 on member MemberName{name=member-256} with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 on shard member-256-shard-default-config14 whose actor address is Actor[akka://cluster-test/user/member-256-shard-default-config#-1046746134] 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config14: Initializing shard [default] 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-256-shard-default-config14 is initialized 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config14: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-256-shard-default-config14, leaderId=member-256-shard-default-config14, leaderPayloadVersion=13] 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config14: Received role changed for member-256-shard-default-config14 from Candidate to Leader 12:23:45,797 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 12:23:45,820 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 12:23:45,825 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2562] 12:23:45,826 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-256}, address: akka://cluster-test@127.0.0.1:2562 12:23:45,826 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 12:23:45,826 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 to akka://cluster-test/user/member-2-shard-default-config 12:23:45,826 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 12:23:46,734 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 12:23:46,735 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:23:46,736 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 12:23:46,736 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 to akka://cluster-test/user/member-256-shard-default-config 12:23:46,736 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 12:23:46,736 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config14: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:46,736 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config14: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2562/user/member-256-shard-default-config#-1046746134, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$fcdTIJnO@3451074e] 12:23:46,737 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 12:23:46,738 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config14: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:46,738 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config14: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2562/user/member-256-shard-default-config#-1046746134, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$fcdTIJnO@3451074e] 12:23:46,739 PM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberWithNameContainedInLeaderIdUnreachable ending 12:23:46,794 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:23:46,794 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 12:23:46,794 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 to akka://cluster-test/user/member-2-shard-default-config 12:23:46,794 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 12:23:46,844 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config14 12:23:46,845 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:23:46,845 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:23:46,852 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:23:46,852 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:23:46,854 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 12:23:46,864 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config14 12:23:46,864 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:23:46,864 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:23:47,805 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - is the new leader among reachable nodes (more leaders may exist) 12:23:47,887 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 12:23:47.889] [cluster-test-akka.actor.internal-dispatcher-3] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:23:47,898 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config15 12:23:47,900 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Recovery complete : shard-manager-config15 12:23:47,900 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config15: Creating local shard: member-1-shard-default-config15 12:23:48,0 PM [main] [INFO] ShardManager - shard-manager-config15: Received role changed for member-1-shard-default-config15 from Follower to Leader 12:23:48,101 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$c] 12:23:48,112 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config15 12:23:48,118 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config16 12:23:48,119 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Recovery complete : shard-manager-config16 12:23:48,119 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config16: Creating local shard: member-1-shard-default-config16 12:23:48,120 PM [main] [INFO] ShardManager - shard-manager-config16: Received role changed for member-1-shard-default-config16 from Candidate to Leader 12:23:48,120 PM [main] [INFO] ShardManager - shard-manager-config16: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config16, leaderId=member-1-shard-default-config16, leaderPayloadVersion=13] 12:23:48,120 PM [main] [INFO] ShardManager - shard-manager-config16: All Shards are ready - data store config16 is ready 12:23:48,222 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$d] 12:23:48,233 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config16 12:23:48,235 PM [main] [INFO] ShardManagerTest - testOnCreateShardWithLocalMemberNotInShardConfig starting 12:23:48,237 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Starting ShardManager shard-manager-config17 12:23:48,239 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config17 12:23:48,239 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:48,253 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config17: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@5b0ccd51] 12:23:48,255 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config17 doCreateShard: shardId: member-1-shard-foo-config17, memberNames: [MemberName{name=member-5}, MemberName{name=member-6}], peerAddresses: {}, isActiveMember: false 12:23:48,257 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config17: findLocalShard : foo 12:23:48,257 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config17: Scheduling 60000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config17, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=false] 12:23:48,269 PM [test-shard-dispatcher-15] [INFO] Shard - Shard created : member-1-shard-foo-config17, persistent : true 12:23:48,285 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$i/member-1-shard-foo-config17/member-1-shard-foo-config17-notifier#1554893001 created and ready for shard:member-1-shard-foo-config17 12:23:48,292 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$i/member-1-shard-foo-config17/shard-member-1:snapshot-read#-682281315] 12:23:48,294 PM [test-shard-dispatcher-15] [INFO] Shard - Starting recovery for member-1-shard-foo-config17 with journal batch size 1 12:23:48,308 PM [test-shard-dispatcher-15] [DEBUG] Shard - member-1-shard-foo-config17: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 12:23:48,310 PM [test-shard-dispatcher-15] [INFO] Shard - member-1-shard-foo-config17: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:23:48,310 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config17: Initializing shard [foo] 12:23:48,310 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-foo-config17 is initialized 12:23:48,310 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-foo-config17 is initialized - notifying 1 OnShardInitialized callbacks 12:23:48,311 PM [main] [INFO] ShardManagerTest - testOnCreateShardWithLocalMemberNotInShardConfig ending 12:23:48,316 PM [test-shard-dispatcher-15] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config17: onLeadershipChange, isLeader: false, hasLeader : false 12:23:48,317 PM [test-shard-dispatcher-15] [DEBUG] Shard - member-1-shard-foo-config17: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config17 is no longer the leader 12:23:48,317 PM [test-akka.actor.default-dispatcher-4] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config17 , received role change from null to Follower 12:23:48,320 PM [test-shard-dispatcher-15] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 12:23:48,322 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config17 , registered listener akka://test/user/$i 12:23:48,322 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config17: Received role changed for member-1-shard-foo-config17 from null to Follower 12:23:48,412 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$i#1249009583] 12:23:48,423 PM [test-shard-dispatcher-15] [INFO] Shard - Stopping Shard member-1-shard-foo-config17 12:23:48,423 PM [test-shard-dispatcher-15] [DEBUG] MessageAssembler - member-1-shard-foo-config17 (Follower): Closing 12:23:48,424 PM [test-shard-dispatcher-15] [DEBUG] Shard - member-1-shard-foo-config17: Aborting 0 pending queued transactions 12:23:48,425 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config17 12:23:48,427 PM [main] [INFO] ShardManagerTest - testServerRemovedShardActorNotRunning starting 12:23:48,429 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config18 12:23:48,430 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config18 12:23:48,430 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: Creating local shard: member-1-shard-default-config18 12:23:48,430 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: Creating local shard: member-1-shard-people-config18 12:23:48,441 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config18: findLocalShard : people 12:23:48,441 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config18: Returning NotInitializedException for shard people 12:23:48,442 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config18: findLocalShard : default 12:23:48,442 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config18: Returning NotInitializedException for shard default 12:23:48,444 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config18 : Local Shard replica for shard default has been removed 12:23:48,444 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config18: persisting the shard list [people] 12:23:48,444 PM [main] [INFO] ShardManagerTest - testServerRemovedShardActorNotRunning ending 12:23:48,453 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config18 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 12:23:48,459 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config18: Successfully deleted prior snapshots 12:23:48,546 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$e] 12:23:48,556 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Stopping ShardManager shard-manager-config18 12:23:48,561 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Starting ShardManager shard-manager-config19 12:23:48,562 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config19 12:23:48,562 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config19: Creating local shard: member-1-shard-default-config19 12:23:48,571 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:48,571 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config19 12:23:48,571 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config19: Initializing shard [default] 12:23:48,571 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-default-config19 is initialized 12:23:48,571 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config19: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 12:23:48,571 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config19: Returning NoShardLeaderException for shard default 12:23:48,673 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$j#-1141230397] 12:23:48,684 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Stopping ShardManager shard-manager-config19 12:23:48,685 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForFollowerShardWithNoInitialLeaderId starting 12:23:48,687 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Starting ShardManager shard-manager-config20 12:23:48,688 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config20 12:23:48,689 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: Creating local shard: member-1-shard-default-config20 12:23:48,698 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:48,698 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config20 12:23:48,698 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config20: Initializing shard [default] 12:23:48,698 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-default-config20 is initialized 12:23:48,698 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - shard-manager-config20: Received role changed for member-1-shard-default-config20 from Candidate to Follower 12:23:48,698 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config20: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 12:23:48,698 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config20: Returning NoShardLeaderException for shard default 12:23:48,699 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - shard-manager-config20: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config20, leaderId=member-1-shard-default-config20, leaderPayloadVersion=13] 12:23:48,699 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - shard-manager-config20: All Shards are ready - data store config20 is ready 12:23:48,699 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config20: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 12:23:48,699 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config20: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#1269209346, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$fcdTIJnO@cd9eb32] 12:23:48,699 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForFollowerShardWithNoInitialLeaderId starting 12:23:48,801 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$k#-1020098457] 12:23:48,811 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Stopping ShardManager shard-manager-config20 12:23:48,815 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Starting ShardManager shard-manager-config21 12:23:48,817 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config21 12:23:48,836 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:48,836 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config21: onAddShardReplica: AddShardReplica[ShardName=astronauts] 12:23:48,837 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config21: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-548#-128119782 for shard member-1-shard-astronauts-config21 12:23:48,848 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config21: onAddShardReplica: AddShardReplica[ShardName=astronauts] 12:23:48,848 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config21: A shard replica operation for astronauts is already in progress 12:23:48,952 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithCandidateShard starting 12:23:48,954 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config22 12:23:48,956 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config22 12:23:48,956 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: Creating local shard: member-1-shard-default-config22 12:23:48,965 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:48,965 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config22 12:23:48,965 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config22: Initializing shard [default] 12:23:48,965 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config22 is initialized 12:23:48,965 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config22: Received role changed for member-1-shard-default-config22 from null to Candidate 12:23:48,965 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config22: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:48,965 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config22: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config22, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Candidate, leaderId=null, activeMember=true] 12:23:50,182 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config22: Received ShardNotInitializedTimeout message for shard default 12:23:50,182 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config22: Returning NoShardLeaderException for shard default 12:23:50,182 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithCandidateShard ending 12:23:50,284 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$l#-1052601492] 12:23:50,295 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config22 12:23:50,297 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithAddServerReplyFailure starting 12:23:50,300 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config23 12:23:50,301 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config23 12:23:50,321 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:50,331 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config23: onAddShardReplica: AddShardReplica[ShardName=astronauts] 12:23:50,332 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config23: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-553#86429489 for shard member-1-shard-astronauts-config23 12:23:50,333 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config23: Received AddServerReply{status=TIMEOUT} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-553#86429489 12:23:50,333 PM [test-akka.actor.default-dispatcher-14] [WARN] ShardManager - shard-manager-config23: Leader failed to add shard replica astronauts with status TIMEOUT 12:23:50,333 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config23: findLocalShard : astronauts 12:23:50,333 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config23: Local shard astronauts not found - shards present: [] 12:23:50,335 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config23: onAddShardReplica: AddShardReplica[ShardName=astronauts] 12:23:50,336 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config23: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-553#86429489 for shard member-1-shard-astronauts-config23 12:23:50,336 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config23: Received AddServerReply{status=NO_LEADER} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-553#86429489 12:23:50,336 PM [test-akka.actor.default-dispatcher-11] [WARN] ShardManager - shard-manager-config23: Leader failed to add shard replica astronauts with status NO_LEADER 12:23:50,337 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithAddServerReplyFailure ending 12:23:50,438 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-astronauts-config#-1938128215] 12:23:50,449 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/shardmanager-config23] 12:23:50,449 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config23 12:23:50,453 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config24 12:23:50,454 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config24 12:23:50,454 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config24: Creating local shard: member-1-shard-default-config24 12:23:50,454 PM [main] [INFO] ShardManager - shard-manager-config24: Received role changed for member-1-shard-default-config24 from Candidate to Follower 12:23:50,454 PM [main] [INFO] ShardManager - shard-manager-config24 Received follower initial sync status for member-1-shard-default-config24 status sync done true 12:23:50,454 PM [main] [INFO] ShardManager - shard-manager-config24 Received follower initial sync status for member-1-shard-default-config24 status sync done false 12:23:50,556 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$f] 12:23:50,567 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config24 12:23:50,571 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config25 12:23:50,573 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Recovery complete : shard-manager-config25 12:23:50,573 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config25: Creating local shard: member-1-shard-default-config25 12:23:50,573 PM [main] [INFO] ShardManager - shard-manager-config25: Received role changed for unknown from Candidate to Leader 12:23:50,675 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$g] 12:23:50,686 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config25 12:23:50,688 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithUninitializedShard starting 12:23:50,690 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config26 12:23:50,691 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config26 12:23:50,691 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config26: Creating local shard: member-1-shard-default-config26 12:23:50,701 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:50,701 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config26 12:23:50,701 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:50,701 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config26, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 12:23:51,311 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: Received ShardNotInitializedTimeout message for shard default 12:23:51,311 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: Returning NotInitializedException for shard default 12:23:51,312 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: Initializing shard [default] 12:23:51,312 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config26 is initialized 12:23:51,512 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithUninitializedShard ending 12:23:51,614 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$m#579502344] 12:23:51,625 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config26 12:23:51,629 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config27 12:23:51,630 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config27 12:23:51,631 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config27: Creating local shard: member-1-shard-default-config27 12:23:51,640 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:51,640 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config27 12:23:51,640 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config27: Initializing shard [default] 12:23:51,640 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config27 is initialized 12:23:51,640 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config27: findLocalShard : default 12:23:51,742 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$n#412888631] 12:23:51,753 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config27 12:23:51,755 PM [main] [INFO] ShardManagerTest - testRegisterForShardLeaderChanges starting 12:23:51,757 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config28 12:23:51,762 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config28 12:23:51,762 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config28: Creating local shard: member-1-shard-default-config28 12:23:51,768 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:51,768 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config28 12:23:51,768 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config28: Initializing shard [default] 12:23:51,768 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config28 is initialized 12:23:51,768 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config28: onRegisterForShardAvailabilityChanges: org.opendaylight.controller.cluster.datastore.shardmanager.RegisterForShardAvailabilityChanges@9982961 12:23:51,769 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=13] 12:23:51,795 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=13] 12:23:52,295 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=null, leaderPayloadVersion=13] 12:23:52,306 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-2-shard-default-config28, leaderPayloadVersion=13] 12:23:52,318 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=13] 12:23:52,818 PM [main] [INFO] ShardManagerTest - testRegisterForShardLeaderChanges ending 12:23:52,919 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$o#-1704481023] 12:23:52,930 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config28 12:23:52,934 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config29 12:23:52,935 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config29 12:23:52,935 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config29: Creating local shard: member-1-shard-default-config29 12:23:53,37 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$h] 12:23:53,47 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config29 12:23:53,52 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Starting ShardManager shard-manager-config30 12:23:53,53 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config30 12:23:53,53 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config30: Creating local shard: member-1-shard-default-config30 12:23:53,62 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:53,62 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config30 12:23:53,62 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config30: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=non-existent, waitUntilReady=false] 12:23:53,62 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config30: No shard found for non-existent 12:23:53,164 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$p#-962107841] 12:23:53,175 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Stopping ShardManager shard-manager-config30 12:23:53,181 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config31 12:23:53,183 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Recovery complete : shard-manager-config31 12:23:53,191 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config31: onAddShardReplica: AddShardReplica[ShardName=model-inventory] 12:23:53,192 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config31: No module configuration exists for shard model-inventory 12:23:53,293 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$q#500212546] 12:23:53,303 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config31 12:23:53,306 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingReplicaInRemoteShardLeader starting 12:23:53,308 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config32 12:23:53,309 PM [test-akka.actor.default-dispatcher-4] [INFO] ShardManager - Recovery complete : shard-manager-config32 12:23:53,309 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config32: Creating local shard: member-1-shard-default-config32 12:23:53,318 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:53,318 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config32 12:23:53,319 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: Initializing shard [default] 12:23:53,319 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config32 is initialized 12:23:53,319 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config32: Received MemberUp: memberName: MemberName{name=leader-member}, address: akka://test 12:23:53,319 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer leader-member-shard-default-config32 with address akka://test/user/shardmanager-config32/leader-member-shard-default-config32 12:23:53,319 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer leader-member-shard-default-config32 with address akka://test/user/shardmanager-config32/leader-member-shard-default-config32 to akka://test/user/member-1-shard-default-config 12:23:53,319 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config32: Received role changed for member-1-shard-default-config32 from Candidate to Follower 12:23:53,319 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config32: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config32, leaderId=leader-member-shard-default-config32, leaderPayloadVersion=13] 12:23:53,319 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config32: All Shards are ready - data store config32 is ready 12:23:53,319 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 12:23:53,320 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:53,320 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=13] 12:23:53,320 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 12:23:53,320 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config32: Received AddServerReply{status=ALREADY_EXISTS} for shard default from leader akka://test/user/shardmanager-config32/leader-member-shard-default-config32 12:23:53,320 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config32: Local shard default already exists 12:23:53,369 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config32: findLocalShard : default 12:23:53,369 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 12:23:53,370 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:53,370 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=13] 12:23:53,370 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 12:23:53,371 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: Received AddServerReply{status=ALREADY_EXISTS} for shard default from leader akka://test/user/shardmanager-config32/leader-member-shard-default-config32 12:23:53,371 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: Local shard default already exists 12:23:53,373 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-default-config32 12:23:53,373 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 12:23:53,373 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:53,373 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=13] 12:23:53,373 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 12:23:53,492 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config32: AddServer request to akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for default failed akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://test/), Path(/user/shardmanager-config32/leader-member-shard-default-config32)]] after [100 ms]. Message of type [org.opendaylight.controller.cluster.raft.messages.AddServer]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. 12:23:53,493 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config32: findLocalShard : default 12:23:53,493 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingReplicaInRemoteShardLeader ending 12:23:53,595 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/shardmanager-config32] 12:23:53,606 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config32 12:23:53,608 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForRemoteShard starting 12:23:53,617 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 12:23:53,626 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [-6693820498481196809] 12:23:53,626 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 12:23:53,629 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 12:23:53,629 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 12:23:53,629 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:53,629 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:53,644 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 12:23:53,644 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 12:23:53,644 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 12:23:53,644 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config33] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:53,649 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config33 12:23:53,650 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 12:23:53,653 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config33 12:23:53,653 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-1-shard-default-config33 12:23:53,653 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 12:23:53,653 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 12:23:53,658 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-1348763505884250093] 12:23:53,659 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 12:23:53,660 PM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:23:53,660 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 12:23:53,671 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:53,672 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:53,680 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config33 12:23:53,684 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config33] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:53,688 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:53,688 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config33 12:23:53,688 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 12:23:53,688 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 to akka://test/user/member-1-shard-default-config 12:23:53,688 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config33: updated peer member-1-shard-default-config33 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 on shard member-1-shard-default-config33 whose actor address is Actor[akka://test/user/member-1-shard-default-config#-1003157343] 12:23:53,690 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Recovery complete : shard-manager-config33 12:23:53,690 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-2-shard-default-config33 12:23:53,690 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-2-shard-astronauts-config33 12:23:53,690 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:53,690 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config33 12:23:53,690 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-2-shard-astronauts-config33 12:23:53,690 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config33: Initializing shard [astronauts] 12:23:53,690 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardInformation - Shard member-2-shard-astronauts-config33 is initialized 12:23:53,691 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config33: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-astronauts-config33, leaderId=member-2-shard-astronauts-config33, leaderPayloadVersion=12] 12:23:53,691 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config33: Received role changed for member-2-shard-astronauts-config33 from Candidate to Leader 12:23:53,722 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 12:23:53,726 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 12:23:53,727 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 12:23:53,727 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 12:23:53,727 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 12:23:53,727 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-astronauts-config33 12:23:53,727 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-astronauts-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 12:23:54,644 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 12:23:54,644 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:23:54,644 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 12:23:54,644 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 to akka://test/user/member-1-shard-default-config 12:23:54,645 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config33: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=false] 12:23:54,645 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config33: findPrimary for astronauts forwarding to remote ShardManager akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33, visitedAddresses: [akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33] 12:23:54,652 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.RemoteFindPrimary [shardName=astronauts, waitUntilReady=false] 12:23:54,652 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: Found primary for astronauts: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config#-526117093, primaryVersion=12] 12:23:54,655 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForRemoteShard ending 12:23:54,699 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:23:54,699 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 12:23:54,699 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 12:23:54,699 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-astronauts-config33 12:23:54,699 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-astronauts-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 12:23:54,756 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config33 12:23:54,757 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:23:54,757 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:23:54,764 PM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:23:54,765 PM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:23:54,768 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 12:23:54,779 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config33 12:23:54,779 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:23:54,779 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:23:55,802 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 12:23:55.803] [cluster-test-akka.actor.internal-dispatcher-7] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:23:55,818 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Starting ShardManager shard-manager-config34 12:23:55,819 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config34 12:23:55,819 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config34: Creating local shard: member-1-shard-default-config34 12:23:55,829 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config34: findLocalShard : default 12:23:55,829 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config34: Returning NotInitializedException for shard default 12:23:55,931 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$r#1004238804] 12:23:55,941 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Stopping ShardManager shard-manager-config34 12:23:55,946 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config35 12:23:55,947 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Recovery complete : shard-manager-config35 12:23:55,948 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config35: Creating local shard: member-1-shard-default-config35 12:23:55,955 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:55,955 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config35 12:23:55,955 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config35: Initializing shard [default] 12:23:55,955 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardInformation - Shard member-1-shard-default-config35 is initialized 12:23:55,956 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config35: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config35, leaderId=member-1-shard-default-config35, leaderPayloadVersion=13] 12:23:55,956 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config35: Received role changed for member-1-shard-default-config35 from Candidate to Leader 12:23:55,956 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config35: All Shards are ready - data store config35 is ready 12:23:55,956 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config35: onChangeShardServersVotingStatus: ChangeShardMembersVotingStatus [shardName=default, meberVotingStatusMap={member-2=true}] 12:23:55,956 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config35: findLocalShard : default 12:23:55,956 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config35: Sending ChangeServersVotingStatus message ChangeServersVotingStatus [serverVotingStatusMap={member-2-shard-default-config35=true}, serversVisited=[]] to local shard akka://test/user/member-1-shard-default-config35 12:23:55,957 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config35: Received ServerChangeReply{status=OK} from local shard akka://test/user/member-1-shard-default-config35 12:23:55,957 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config35: ChangeServersVotingStatus succeeded for shard default 12:23:56,107 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config35#-1847775337] 12:23:56,120 PM [main] [INFO] ShardManagerTest - testShutDown starting 12:23:56,123 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Starting ShardManager shard-manager-config36 12:23:56,124 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config36 12:23:56,124 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config36: Creating local shard: member-1-shard-shard2-config36 12:23:56,125 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config36: Creating local shard: member-1-shard-shard1-config36 12:23:56,134 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:56,134 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config36 12:23:56,134 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config36 12:23:56,134 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: Initializing shard [shard1] 12:23:56,134 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config36 is initialized 12:23:56,134 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: Initializing shard [shard2] 12:23:56,134 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config36 is initialized 12:23:56,134 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: Issuing gracefulStop to shard member-1-shard-shard2-config36 12:23:56,135 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: Issuing gracefulStop to shard member-1-shard-shard1-config36 12:23:56,136 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Shutting down ShardManager shard-manager-config36 - waiting on 2 shards 12:23:56,686 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-shard1-config36#-1949652258] 12:23:56,687 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-shard2-config36#1560183060] 12:23:56,687 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: All shards shutdown - sending PoisonPill to self 12:23:56,688 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config36 12:23:56,688 PM [main] [INFO] ShardManagerTest - testShutDown ending 12:23:56,789 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$t#-1783971804] 12:23:56,802 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShardBeforeMemberUp starting 12:23:56,803 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config37 12:23:56,805 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config37 12:23:56,806 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config37: Creating local shard: member-1-shard-default-config37 12:23:56,814 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:56,814 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config37 12:23:56,814 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config37: Initializing shard [default] 12:23:56,814 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config37 is initialized 12:23:56,814 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config37: Received role changed for member-1-shard-default-config37 from Candidate to Follower 12:23:56,814 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerTest$TestShardManager - Received unhandled message LeaderStateChanged [memberId=member-1-shard-default-config37, leaderId=member-2-shard-default-config37, leaderPayloadVersion=13] 12:23:56,816 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config37: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 12:23:56,816 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config37: Returning NoShardLeaderException for shard default 12:23:56,816 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShardBeforeMemberUp ending 12:23:56,917 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$u#42404923] 12:23:56,928 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config37 12:23:56,930 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithNoRoleShard starting 12:23:56,932 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config38 12:23:56,933 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config38 12:23:56,933 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config38: Creating local shard: member-1-shard-default-config38 12:23:56,943 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:56,943 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config38 12:23:56,943 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: Initializing shard [default] 12:23:56,943 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config38 is initialized 12:23:56,943 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:56,943 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config38, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 12:23:58,161 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: Received ShardNotInitializedTimeout message for shard default 12:23:58,162 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: Returning NoShardLeaderException for shard default 12:23:58,162 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithNoRoleShard ending 12:23:58,264 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$v#2044815260] 12:23:58,275 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config38 12:23:58,276 PM [main] [INFO] ShardManagerTest - testShardPersistenceWithRestoredData starting 12:23:58,279 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config39 12:23:58,281 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: onSnapshotOffer: ShardManagerSnapshot [ShardList = [default, astronauts] ] 12:23:58,281 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: adding shard astronauts 12:23:58,281 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: removing shard people 12:23:58,282 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config39 12:23:58,282 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config39: Creating local shard: member-1-shard-default-config39 12:23:58,282 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config39: Creating local shard: member-1-shard-astronauts-config39 12:23:58,293 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : people 12:23:58,293 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config39: Local shard people not found - shards present: [default, astronauts] 12:23:58,293 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : default 12:23:58,293 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config39: Returning NotInitializedException for shard default 12:23:58,293 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : astronauts 12:23:58,293 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config39: Returning NotInitializedException for shard astronauts 12:23:58,293 PM [main] [INFO] ShardManagerTest - testShardPersistenceWithRestoredData ending 12:23:58,395 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$i] 12:23:58,406 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Stopping ShardManager shard-manager-config39 12:23:58,407 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShard starting 12:23:58,410 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config40 12:23:58,411 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Recovery complete : shard-manager-config40 12:23:58,411 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config40: Creating local shard: member-1-shard-default-config40 12:23:58,420 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config40 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config40: Initializing shard [default] 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config40 is initialized 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config40: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config40 with address akka://test/user/shardmanager-config40/member-2-shard-default-config40 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config40 with address akka://test/user/shardmanager-config40/member-2-shard-default-config40 to akka://test/user/member-1-shard-default-config 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config40: Received role changed for member-1-shard-default-config40 from Candidate to Follower 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config40: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config40, leaderId=member-2-shard-default-config40, leaderPayloadVersion=12] 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config40: All Shards are ready - data store config40 is ready 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config40: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 12:23:58,421 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config40: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config40/member-2-shard-default-config40, primaryVersion=12] 12:23:58,421 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShard ending 12:23:58,523 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$w#2110850835] 12:23:58,533 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config40 12:23:58,536 PM [main] [INFO] ShardManagerTest - testShardAvailabilityOnChangeOfMemberReachability starting 12:23:58,542 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 12:23:58,562 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [914238651878669415] 12:23:58,562 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 12:23:58,565 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 12:23:58,565 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 12:23:58,572 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:58,572 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:58,580 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 12:23:58,580 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 12:23:58,580 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 12:23:58,580 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config41] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:58,581 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config41 12:23:58,604 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 12:23:58,614 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config41 12:23:58,614 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config41: Creating local shard: member-1-shard-default-config41 12:23:58,614 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 12:23:58,614 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 12:23:58,634 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-6282257921698366057] 12:23:58,634 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 12:23:58,641 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:23:58,641 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 12:23:58,654 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:23:58,654 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:23:58,658 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config41] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:23:58,660 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:58,660 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config41 12:23:58,660 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 12:23:58,660 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 12:23:58,660 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config41: updated peer member-1-shard-default-config41 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 on shard member-1-shard-default-config41 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#-1953447335] 12:23:58,660 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config41: Initializing shard [default] 12:23:58,660 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is initialized 12:23:58,660 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config41: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config41, leaderId=member-2-shard-default-config41, leaderPayloadVersion=13] 12:23:58,660 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config41: Received role changed for member-1-shard-default-config41 from Candidate to Follower 12:23:58,664 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config41 12:23:58,679 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config41 12:23:58,679 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Creating local shard: member-2-shard-default-config41 12:23:58,679 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:23:58,679 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config41 12:23:58,679 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Initializing shard [default] 12:23:58,679 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-2-shard-default-config41 is initialized 12:23:58,679 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config41: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config41, leaderId=member-2-shard-default-config41, leaderPayloadVersion=13] 12:23:58,679 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config41: Received role changed for member-2-shard-default-config41 from Candidate to Leader 12:23:58,679 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 12:23:58,722 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 12:23:58,727 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 12:23:58,728 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 12:23:58,728 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 12:23:58,728 PM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 to akka://cluster-test/user/member-2-shard-default-config 12:23:58,728 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 12:23:59,582 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 12:23:59,582 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:23:59,582 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 12:23:59,582 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 12:23:59,582 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 12:23:59,583 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:59,583 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=13] 12:23:59,583 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 12:23:59,583 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as unavailable. 12:23:59,584 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received MemberRemoved: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 12:23:59,584 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:23:59,584 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config41, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config41, activeMember=true] 12:23:59,673 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:23:59,673 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 12:23:59,673 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-2-shard-default-config 12:23:59,673 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 12:24:00,800 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config41: Received ShardNotInitializedTimeout message for shard default 12:24:00,800 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config41: Returning NoShardLeaderException for shard default 12:24:00,801 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Received ReachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 12:24:00,801 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 12:24:00,801 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 12:24:00,801 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as available. 12:24:00,801 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:24:00,801 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=13] 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as unavailable. 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config41, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config41, activeMember=true] 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Received ReachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is initialized - notifying 1 OnShardInitialized callbacks 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as available. 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is ready - notifying 1 OnShardInitialized callbacks 12:24:00,802 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=13] 12:24:00,803 PM [main] [INFO] ShardManagerTest - testShardAvailabilityOnChangeOfMemberReachability ending 12:24:00,904 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shard-manager-config41 12:24:00,905 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:24:00,905 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:24:00,913 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:24:00,913 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:24:00,915 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 12:24:00,922 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config41 12:24:00,928 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:24:00,928 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [WARN] [03/13/2024 12:24:01.945] [cluster-test-akka.actor.internal-dispatcher-3] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:24:01,966 PM [main] [INFO] ShardManagerTest - testAddShardReplica starting 12:24:01,978 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 12:24:01,986 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [7529700413024178732] 12:24:01,986 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 12:24:01,993 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 12:24:01,993 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 12:24:01,993 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:24:01,994 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:24:02,4 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 12:24:02,4 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 12:24:02,4 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 12:24:02,6 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config42] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:24:02,10 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Starting ShardManager shardmanager-config42 12:24:02,42 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 12:24:02,60 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Recovery complete : shardmanager-config42 12:24:02,60 PM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-1-shard-default-config42 12:24:02,60 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 12:24:02,60 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 12:24:02,63 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [7713616710567144474] 12:24:02,63 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 12:24:02,69 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:24:02,69 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 12:24:02,74 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:24:02,74 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:24:02,86 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shardmanager-config42 12:24:02,87 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config42] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:24:02,90 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:02,90 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config42 12:24:02,90 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 12:24:02,90 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 to akka://cluster-test/user/member-1-shard-default-config 12:24:02,90 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42: updated peer member-1-shard-default-config42 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 on shard member-1-shard-default-config42 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#524253487] 12:24:02,107 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shardmanager-config42 12:24:02,107 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-2-shard-default-config42 12:24:02,108 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-2-shard-astronauts-config42 12:24:02,108 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:02,108 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config42 12:24:02,108 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-2-shard-astronauts-config42 12:24:02,108 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: Initializing shard [astronauts] 12:24:02,108 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-2-shard-astronauts-config42 is initialized 12:24:02,108 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shardmanager-config42: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-astronauts-config42, leaderId=member-2-shard-astronauts-config42, leaderPayloadVersion=12] 12:24:02,108 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shardmanager-config42: Received role changed for member-2-shard-astronauts-config42 from Candidate to Leader 12:24:02,165 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 12:24:02,169 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 12:24:02,170 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 12:24:02,170 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 12:24:02,170 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 12:24:02,170 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-astronauts-config42 12:24:02,170 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-astronauts-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 12:24:03,11 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 12:24:03,11 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:24:03,11 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 12:24:03,11 PM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 to akka://cluster-test/user/member-1-shard-default-config 12:24:03,14 PM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shardmanager-config42: onAddShardReplica: AddShardReplica[ShardName=astronauts] 12:24:03,15 PM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shardmanager-config42: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=true] 12:24:03,15 PM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shardmanager-config42: findPrimary for astronauts forwarding to remote ShardManager akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42, visitedAddresses: [akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42] 12:24:03,18 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.RemoteFindPrimary [shardName=astronauts, waitUntilReady=true] 12:24:03,18 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: Found primary for astronauts: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config, primaryVersion=12] 12:24:03,20 PM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shardmanager-config42: Sending AddServer message to peer akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config for shard member-1-shard-astronauts-config42 12:24:03,30 PM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shardmanager-config42: Received AddServerReply{status=OK, leaderHint=member-2-shard-astronauts-config42} for shard astronauts from leader akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config 12:24:03,30 PM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shardmanager-config42: Leader shard successfully added the replica shard astronauts 12:24:03,30 PM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-astronauts-config42 12:24:03,30 PM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shardmanager-config42: persisting the shard list [default, astronauts] 12:24:03,31 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shardmanager-config42 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 12:24:03,32 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shardmanager-config42: Successfully deleted prior snapshots 12:24:03,65 PM [main] [INFO] ShardManagerTest - testAddShardReplica ending 12:24:03,95 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 12:24:03,95 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 12:24:03,95 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 12:24:03,95 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-astronauts-config42 12:24:03,95 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-astronauts-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 12:24:03,167 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shardmanager-config42 12:24:03,168 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:24:03,168 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:24:03,179 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:24:03,179 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 12:24:03,180 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 12:24:03,191 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shardmanager-config42 12:24:03,192 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:24:03,192 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 12:24:04,209 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 12:24:04.211] [cluster-test-akka.actor.internal-dispatcher-2] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 12:24:04,228 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config43 12:24:04,228 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config43 12:24:04,229 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Creating local shard: member-1-shard-default-config43 12:24:04,232 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:04,232 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config43 12:24:04,232 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Initializing shard [default] 12:24:04,232 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config43 is initialized 12:24:04,232 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config43: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config43, leaderId=member-1-shard-default-config43, leaderPayloadVersion=13] 12:24:04,232 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config43: Received role changed for member-1-shard-default-config43 from Candidate to Leader 12:24:04,232 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config43: All Shards are ready - data store config43 is ready 12:24:04,232 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: onRemoveShardReplica: RemoveShardReplica [shardName=default, memberName=MemberName{name=member-1}] 12:24:04,232 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:24:04,232 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config43#2142594625, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$fcdTIJnO@65a388a0] 12:24:04,233 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Sending RemoveServer message to peer akka://test@127.0.0.1:2565/user/member-1-shard-default-config43#2142594625 for shard member-1-shard-default-config43 12:24:04,233 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config43: Received RemoveServerReply{status=OK} for shard default 12:24:04,233 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config43: Leader shard successfully removed the replica shard default 12:24:04,384 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config43#2142594625] 12:24:04,396 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithIsolatedLeaderShard starting 12:24:04,399 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config44 12:24:04,400 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config44 12:24:04,400 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config44: Creating local shard: member-1-shard-default-config44 12:24:04,409 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:04,409 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config44 12:24:04,409 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: Initializing shard [default] 12:24:04,409 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config44 is initialized 12:24:04,409 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config44: Received role changed for member-1-shard-default-config44 from null to IsolatedLeader 12:24:04,409 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 12:24:04,409 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config44, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=IsolatedLeader, leaderId=null, activeMember=true] 12:24:05,622 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: Received ShardNotInitializedTimeout message for shard default 12:24:05,622 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: Returning NoShardLeaderException for shard default 12:24:05,623 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithIsolatedLeaderShard ending 12:24:05,724 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$y#1165684951] 12:24:05,735 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config44 12:24:05,739 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config45 12:24:05,740 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config45 12:24:05,740 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config45: Creating local shard: member-1-shard-default-config45 12:24:05,749 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:05,749 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config45 12:24:05,749 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config45: findLocalShard : non-existent 12:24:05,749 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config45: Local shard non-existent not found - shards present: [default] 12:24:05,851 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$z#-1857866008] 12:24:05,861 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Stopping ShardManager shard-manager-config45 12:24:05,866 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Starting ShardManager shard-manager-config46 12:24:05,867 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Recovery complete : shard-manager-config46 12:24:05,888 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:05,888 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config46: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-2}] 12:24:05,889 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config46: Sending RemoveServer message to peer akka://test@127.0.0.1:2565/system/testProbe-595#-1190685353 for shard member-2-shard-astronauts-config46 12:24:05,900 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config46: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-3}] 12:24:05,900 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config46: A shard replica operation for astronauts is already in progress 12:24:06,6 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Starting ShardManager shard-manager-config47 12:24:06,7 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config47 12:24:06,7 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config47: Creating local shard: member-1-shard-default-config47 12:24:06,16 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config47: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 12:24:06,16 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config47: Returning NotInitializedException for shard default 12:24:06,118 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$A#-1231024318] 12:24:06,129 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Stopping ShardManager shard-manager-config47 12:24:06,130 PM [main] [INFO] ShardManagerTest - testServerRemovedShardActorRunning starting 12:24:06,134 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Starting ShardManager shard-manager-config48 12:24:06,140 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config48 12:24:06,140 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48: Creating local shard: member-1-shard-default-config48 12:24:06,140 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48: Creating local shard: member-1-shard-people-config48 12:24:06,151 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:06,151 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config48 12:24:06,151 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-1-shard-people-config48 12:24:06,152 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config48: Initializing shard [default] 12:24:06,152 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardInformation - Shard member-1-shard-default-config48 is initialized 12:24:06,152 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config48: findLocalShard : people 12:24:06,152 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config48: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-people-config48, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 12:24:06,155 PM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-people-config48, persistent : true 12:24:06,158 PM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$j/member-1-shard-people-config48/member-1-shard-people-config48-notifier#-31584727 created and ready for shard:member-1-shard-people-config48 12:24:06,158 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$j/member-1-shard-people-config48/shard-member-1:snapshot-read#1100526955] 12:24:06,158 PM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-people-config48 with journal batch size 1 12:24:06,159 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-people-config48: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 12:24:06,159 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-people-config48: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:24:06,160 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-people-config48: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,160 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-people-config48: onStateChanged: Closing all transaction chains because shard member-1-shard-people-config48 is no longer the leader 12:24:06,160 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config48: Initializing shard [people] 12:24:06,160 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardInformation - Shard member-1-shard-people-config48 is initialized 12:24:06,160 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-people-config48 , received role change from null to Follower 12:24:06,160 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardInformation - Shard member-1-shard-people-config48 is initialized - notifying 1 OnShardInitialized callbacks 12:24:06,160 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config48: findLocalShard : default 12:24:06,160 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config48 : Sending Shutdown to Shard actor Actor[akka://test/user/member-1-shard-default-config48#1877635832] with 10000 ms timeout 12:24:06,162 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config48 : Local Shard replica for shard default has been removed 12:24:06,162 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config48: persisting the shard list [people] 12:24:06,162 PM [main] [INFO] ShardManagerTest - testServerRemovedShardActorRunning ending 12:24:06,163 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config48 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 12:24:06,163 PM [test-akka.actor.default-dispatcher-17] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-people-config48 , registered listener akka://test/user/$$j 12:24:06,163 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config48: Successfully deleted prior snapshots 12:24:06,163 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config48: Received role changed for member-1-shard-people-config48 from null to Follower 12:24:06,264 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config48#1877635832] 12:24:06,275 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48 : Successfully shut down Shard actor Actor[akka://test/user/member-1-shard-default-config48#1877635832] 12:24:06,275 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$j] 12:24:06,276 PM [test-shard-dispatcher-18] [INFO] Shard - Stopping Shard member-1-shard-people-config48 12:24:06,276 PM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-people-config48 (Follower): Closing 12:24:06,276 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-people-config48: Aborting 0 pending queued transactions 12:24:06,276 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config48 12:24:06,280 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config49 12:24:06,282 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config49 12:24:06,282 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config49: Creating local shard: member-1-shard-default-config49 12:24:06,282 PM [main] [INFO] ShardManager - shard-manager-config49: Received role changed for member-1-shard-default-config49 from null to Follower 12:24:06,283 PM [main] [INFO] ShardManager - shard-manager-config49: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config49, leaderId=member-2-shard-default-config49, leaderPayloadVersion=13] 12:24:06,293 PM [main] [INFO] ShardManager - shard-manager-config49: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 12:24:06,293 PM [main] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config49 with address akka://test/user/shardmanager-config49/member-2-shard-default-config49 12:24:06,293 PM [main] [INFO] ShardManager - shard-manager-config49: All Shards are ready - data store config49 is ready 12:24:06,394 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$k] 12:24:06,405 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config49 12:24:06,407 PM [main] [INFO] ShardManagerTest - testRestoreFromSnapshot starting 12:24:06,410 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config50 12:24:06,411 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config50 12:24:06,411 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Restoring from ShardManagerSnapshot: ShardManagerSnapshot [ShardList = [shard1, shard2, astronauts] ] 12:24:06,411 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: onSnapshotOffer: ShardManagerSnapshot [ShardList = [shard1, shard2, astronauts] ] 12:24:06,411 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: adding shard shard1 12:24:06,411 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: adding shard shard2 12:24:06,411 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: adding shard astronauts 12:24:06,411 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-shard2-config50 12:24:06,412 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-astronauts-config50 12:24:06,412 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-shard1-config50 12:24:06,510 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:06,510 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config50 12:24:06,510 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-astronauts-config50 12:24:06,511 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config50 12:24:06,511 PM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-shard2-config50, persistent : true 12:24:06,512 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-shard2-config50/member-1-shard-shard2-config50-notifier#481730453 created and ready for shard:member-1-shard-shard2-config50 12:24:06,512 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-shard2-config50/shard-member-1:snapshot-read#-2112488212] 12:24:06,512 PM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-shard2-config50 with journal batch size 1 12:24:06,513 PM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-astronauts-config50, persistent : true 12:24:06,513 PM [test-shard-dispatcher-20] [INFO] Shard - Shard created : member-1-shard-shard1-config50, persistent : true 12:24:06,514 PM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-astronauts-config50 with journal batch size 1 12:24:06,514 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-shard1-config50/shard-member-1:snapshot-read#981675535] 12:24:06,514 PM [test-shard-dispatcher-20] [INFO] Shard - Starting recovery for member-1-shard-shard1-config50 with journal batch size 1 12:24:06,514 PM [test-akka.actor.default-dispatcher-16] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-shard1-config50/member-1-shard-shard1-config50-notifier#-2110119083 created and ready for shard:member-1-shard-shard1-config50 12:24:06,515 PM [test-akka.actor.default-dispatcher-16] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-astronauts-config50/member-1-shard-astronauts-config50-notifier#-1030777316 created and ready for shard:member-1-shard-astronauts-config50 12:24:06,515 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 12:24:06,515 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard2-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:24:06,515 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,515 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config50 is no longer the leader 12:24:06,516 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [shard2] 12:24:06,516 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config50 is initialized 12:24:06,516 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from null to Follower 12:24:06,517 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50 (Follower): Received TimeoutNow - switching to Candidate 12:24:06,519 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 12:24:06,519 PM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-shard1-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:24:06,519 PM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,519 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [shard1] 12:24:06,519 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config50 is no longer the leader 12:24:06,519 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config50 is initialized 12:24:06,519 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from null to Follower 12:24:06,519 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config50 (Follower): Received TimeoutNow - switching to Candidate 12:24:06,520 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 12:24:06,520 PM [test-shard-dispatcher-23] [INFO] Shard - member-1-shard-astronauts-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:24:06,520 PM [test-shard-dispatcher-23] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,520 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config50 is no longer the leader 12:24:06,520 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50 (Candidate): Election: Candidate has following voting peers: [] 12:24:06,520 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50: Set currentTerm=1, votedFor=member-1-shard-shard2-config50 12:24:06,520 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50 (Follower): Received TimeoutNow - switching to Candidate 12:24:06,521 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50 (Candidate): Election: Candidate has following voting peers: [] 12:24:06,521 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50: Set currentTerm=1, votedFor=member-1-shard-astronauts-config50 12:24:06,521 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [astronauts] 12:24:06,521 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config50 is initialized 12:24:06,521 PM [test-akka.actor.default-dispatcher-4] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from null to Follower 12:24:06,521 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : shard1 12:24:06,521 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config50 (Candidate): Election: Candidate has following voting peers: [] 12:24:06,521 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : shard2 12:24:06,521 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config50: Set currentTerm=1, votedFor=member-1-shard-shard1-config50 12:24:06,521 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : astronauts 12:24:06,523 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-astronauts-config50/shard-member-1:snapshot-read#-2015186844] 12:24:06,523 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard2-config50 (Candidate): Starting new election term 1 12:24:06,523 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard2-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 12:24:06,523 PM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-shard2-config50 (Follower): Closing 12:24:06,523 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,523 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config50 is no longer the leader 12:24:06,523 PM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-shard1-config50 (Candidate): Starting new election term 1 12:24:06,523 PM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-shard1-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 12:24:06,523 PM [test-shard-dispatcher-20] [DEBUG] MessageAssembler - member-1-shard-shard1-config50 (Follower): Closing 12:24:06,523 PM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,524 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config50 is no longer the leader 12:24:06,524 PM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from Follower to Candidate 12:24:06,524 PM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from Follower to Candidate 12:24:06,532 PM [test-shard-dispatcher-23] [INFO] Shard - member-1-shard-astronauts-config50 (Candidate): Starting new election term 1 12:24:06,532 PM [test-shard-dispatcher-23] [INFO] Shard - member-1-shard-astronauts-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 12:24:06,532 PM [test-shard-dispatcher-23] [DEBUG] MessageAssembler - member-1-shard-astronauts-config50 (Follower): Closing 12:24:06,532 PM [test-shard-dispatcher-23] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,532 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config50 is no longer the leader 12:24:06,532 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: onGetSnapshot 12:24:06,533 PM [test-akka.actor.default-dispatcher-17] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from Follower to Candidate 12:24:06,534 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Expecting 3 shard snapshot replies 12:24:06,535 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50 (Candidate): Received ElectionTimeout 12:24:06,541 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50 (Candidate): Received ElectionTimeout 12:24:06,541 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50 (Candidate): Received ElectionTimeout 12:24:06,544 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50 (Leader): Election: Leader has following peers: [] 12:24:06,545 PM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-shard2-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 12:24:06,545 PM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: true, hasLeader : true 12:24:06,545 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50 (Leader): Election: Leader has following peers: [] 12:24:06,545 PM [test-shard-dispatcher-23] [INFO] Shard - member-1-shard-astronauts-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 12:24:06,545 PM [test-shard-dispatcher-23] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: true, hasLeader : true 12:24:06,546 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50: became leader with frontend state for [] 12:24:06,546 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50: became leader with frontend state for [] 12:24:06,547 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from Candidate to Leader 12:24:06,547 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from Candidate to Leader 12:24:06,549 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50 (Leader): Election: Leader has following peers: [] 12:24:06,549 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard1-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 12:24:06,549 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: true, hasLeader : true 12:24:06,549 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50: became leader with frontend state for [] 12:24:06,549 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from Candidate to Leader 12:24:06,553 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50: onGetSnapshot 12:24:06,553 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50: onGetSnapshot 12:24:06,553 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50: onGetSnapshot 12:24:06,553 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , registered listener akka://test/user/$$l 12:24:06,553 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 12:24:06,553 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 12:24:06,553 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 12:24:06,553 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard1-config50, leaderId=member-1-shard-shard1-config50, leaderPayloadVersion=13] 12:24:06,553 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , registered listener akka://test/user/$$l 12:24:06,553 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , registered listener akka://test/user/$$l 12:24:06,553 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-shard1-config50 from Candidate to Leader 12:24:06,553 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard2-config50, leaderId=member-1-shard-shard2-config50, leaderPayloadVersion=13] 12:24:06,554 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-shard2-config50 from Candidate to Leader 12:24:06,554 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-astronauts-config50, leaderId=member-1-shard-astronauts-config50, leaderPayloadVersion=13] 12:24:06,554 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-astronauts-config50 from Candidate to Leader 12:24:06,554 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: All Shards are ready - data store config50 is ready 12:24:06,559 PM [test-shard-dispatcher-23] [DEBUG] Shard - member-1-shard-astronauts-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 12:24:06,559 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 12:24:06,559 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 12:24:06,565 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-shard2-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@69a6ea4e, electionTerm=1, electionVotedFor=member-1-shard-shard2-config50, ServerConfigPayload=null]] 12:24:06,566 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-astronauts-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@6ca846f8, electionTerm=1, electionVotedFor=member-1-shard-astronauts-config50, ServerConfigPayload=null]] 12:24:06,566 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-shard1-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@328235b9, electionTerm=1, electionVotedFor=member-1-shard-shard1-config50, ServerConfigPayload=null]] 12:24:06,566 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: All shard snapshots received 12:24:06,567 PM [main] [INFO] ShardManagerTest - testRestoreFromSnapshot ending 12:24:06,669 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$l] 12:24:06,680 PM [test-shard-dispatcher-20] [INFO] Shard - Stopping Shard member-1-shard-shard1-config50 12:24:06,680 PM [test-shard-dispatcher-18] [INFO] Shard - Stopping Shard member-1-shard-shard2-config50 12:24:06,680 PM [test-shard-dispatcher-18] [DEBUG] MessageSlicer - member-1-shard-shard2-config50 (Leader)_slicer-id-214: Closing 12:24:06,680 PM [test-shard-dispatcher-20] [DEBUG] MessageSlicer - member-1-shard-shard1-config50 (Leader)_slicer-id-216: Closing 12:24:06,680 PM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-astronauts-config50 12:24:06,680 PM [test-shard-dispatcher-19] [DEBUG] MessageSlicer - member-1-shard-astronauts-config50 (Leader)_slicer-id-215: Closing 12:24:06,680 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50: Aborting 0 pending queued transactions 12:24:06,680 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config50: Aborting 0 pending queued transactions 12:24:06,680 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: Aborting 0 pending queued transactions 12:24:06,681 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Stopping ShardManager shard-manager-config50 12:24:06,683 PM [main] [INFO] ShardManagerTest - testOnCreateShardWithNoInitialSchemaContext starting 12:24:06,685 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Starting ShardManager shard-manager-config51 12:24:06,686 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Recovery complete : shard-manager-config51 12:24:06,696 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config51: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@60c40b6f] 12:24:06,696 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config51 doCreateShard: shardId: member-1-shard-foo-config51, memberNames: [MemberName{name=member-1}], peerAddresses: {}, isActiveMember: true 12:24:06,696 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:06,696 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - Creating Shard member-1-shard-foo-config51 12:24:06,697 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config51: findLocalShard : foo 12:24:06,697 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config51: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config51, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 12:24:06,697 PM [test-shard-dispatcher-19] [INFO] Shard - Shard created : member-1-shard-foo-config51, persistent : true 12:24:06,698 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$B/member-1-shard-foo-config51/shard-member-1:snapshot-read#-2145951454] 12:24:06,698 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$B/member-1-shard-foo-config51/member-1-shard-foo-config51-notifier#1433446896 created and ready for shard:member-1-shard-foo-config51 12:24:06,698 PM [test-shard-dispatcher-19] [INFO] Shard - Starting recovery for member-1-shard-foo-config51 with journal batch size 1 12:24:06,702 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config51: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 12:24:06,702 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-foo-config51: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:24:06,702 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config51: Initializing shard [foo] 12:24:06,702 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,702 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardInformation - Shard member-1-shard-foo-config51 is initialized 12:24:06,702 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config51: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config51 is no longer the leader 12:24:06,702 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardInformation - Shard member-1-shard-foo-config51 is initialized - notifying 1 OnShardInitialized callbacks 12:24:06,702 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config51 (Follower): Received TimeoutNow - switching to Candidate 12:24:06,702 PM [main] [INFO] ShardManagerTest - testOnCreateShardWithNoInitialSchemaContext ending 12:24:06,702 PM [test-akka.actor.default-dispatcher-16] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from null to Follower 12:24:06,702 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config51 (Candidate): Election: Candidate has following voting peers: [] 12:24:06,702 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config51: Set currentTerm=1, votedFor=member-1-shard-foo-config51 12:24:06,702 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-foo-config51 (Candidate): Starting new election term 1 12:24:06,702 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-foo-config51 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 12:24:06,702 PM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-foo-config51 (Follower): Closing 12:24:06,702 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,702 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config51: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config51 is no longer the leader 12:24:06,702 PM [test-akka.actor.default-dispatcher-17] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from Follower to Candidate 12:24:06,703 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config51 (Candidate): Received ElectionTimeout 12:24:06,703 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config51 (Leader): Election: Leader has following peers: [] 12:24:06,703 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-foo-config51 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 12:24:06,703 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: true, hasLeader : true 12:24:06,703 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config51: became leader with frontend state for [] 12:24:06,703 PM [test-akka.actor.default-dispatcher-17] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from Candidate to Leader 12:24:06,704 PM [test-akka.actor.default-dispatcher-17] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , registered listener akka://test/user/$B 12:24:06,704 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config51: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-foo-config51, leaderId=member-1-shard-foo-config51, leaderPayloadVersion=13] 12:24:06,704 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config51: Received role changed for member-1-shard-foo-config51 from Candidate to Leader 12:24:06,704 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config51: All Shards are ready - data store config51 is ready 12:24:06,803 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$B#-1312613605] 12:24:06,815 PM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-foo-config51 12:24:06,815 PM [test-shard-dispatcher-19] [DEBUG] MessageSlicer - member-1-shard-foo-config51 (Leader)_slicer-id-218: Closing 12:24:06,815 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config51: Aborting 0 pending queued transactions 12:24:06,816 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Stopping ShardManager shard-manager-config51 12:24:06,818 PM [main] [INFO] ShardManagerTest - testGetSnapshot starting 12:24:06,820 PM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Starting ShardManager shard-manager-config52 12:24:06,823 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - Recovery complete : shard-manager-config52 12:24:06,823 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config52: Creating local shard: member-1-shard-shard2-config52 12:24:06,824 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config52: Creating local shard: member-1-shard-shard1-config52 12:24:06,831 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 12:24:06,832 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:06,832 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config52 12:24:06,832 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config52 12:24:06,832 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : shard1 12:24:06,832 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config52: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-shard1-config52, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 12:24:06,833 PM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-shard1-config52, persistent : true 12:24:06,833 PM [test-shard-dispatcher-19] [INFO] Shard - Shard created : member-1-shard-shard2-config52, persistent : true 12:24:06,834 PM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-shard1-config52 with journal batch size 1 12:24:06,834 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-shard2-config52/shard-member-1:snapshot-read#573983913] 12:24:06,834 PM [test-shard-dispatcher-19] [INFO] Shard - Starting recovery for member-1-shard-shard2-config52 with journal batch size 1 12:24:06,834 PM [test-akka.actor.default-dispatcher-17] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-shard2-config52/member-1-shard-shard2-config52-notifier#1001723213 created and ready for shard:member-1-shard-shard2-config52 12:24:06,834 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-shard1-config52/shard-member-1:snapshot-read#-480520314] 12:24:06,834 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-shard1-config52/member-1-shard-shard1-config52-notifier#37985051 created and ready for shard:member-1-shard-shard1-config52 12:24:06,837 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 12:24:06,837 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard1-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:24:06,837 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 12:24:06,837 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard2-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:24:06,837 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,837 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config52 is no longer the leader 12:24:06,837 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [shard1] 12:24:06,837 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config52 is initialized 12:24:06,837 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,837 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config52 is initialized - notifying 1 OnShardInitialized callbacks 12:24:06,837 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config52 is no longer the leader 12:24:06,837 PM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from null to Follower 12:24:06,837 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52 (Follower): Received TimeoutNow - switching to Candidate 12:24:06,838 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52 (Candidate): Election: Candidate has following voting peers: [] 12:24:06,838 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: Set currentTerm=1, votedFor=member-1-shard-shard1-config52 12:24:06,838 PM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from null to Follower 12:24:06,838 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard1-config52 (Candidate): Starting new election term 1 12:24:06,838 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52 (Follower): Received TimeoutNow - switching to Candidate 12:24:06,838 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard1-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 12:24:06,838 PM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-shard1-config52 (Follower): Closing 12:24:06,838 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [shard2] 12:24:06,838 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,838 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52 (Candidate): Election: Candidate has following voting peers: [] 12:24:06,838 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config52 is initialized 12:24:06,838 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config52 is no longer the leader 12:24:06,838 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52: Set currentTerm=1, votedFor=member-1-shard-shard2-config52 12:24:06,838 PM [test-akka.actor.default-dispatcher-4] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from Follower to Candidate 12:24:06,838 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard2-config52 (Candidate): Starting new election term 1 12:24:06,838 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard2-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 12:24:06,838 PM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-shard2-config52 (Follower): Closing 12:24:06,838 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,838 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config52 is no longer the leader 12:24:06,838 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from Follower to Candidate 12:24:06,838 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : shard2 12:24:06,838 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 12:24:06,838 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Expecting 2 shard snapshot replies 12:24:06,839 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52 (Candidate): Received ElectionTimeout 12:24:06,839 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52 (Candidate): Received ElectionTimeout 12:24:06,839 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52 (Leader): Election: Leader has following peers: [] 12:24:06,839 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard1-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 12:24:06,839 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: true, hasLeader : true 12:24:06,839 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52 (Leader): Election: Leader has following peers: [] 12:24:06,839 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard2-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 12:24:06,839 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: true, hasLeader : true 12:24:06,839 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: became leader with frontend state for [] 12:24:06,840 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: became leader with frontend state for [] 12:24:06,840 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from Candidate to Leader 12:24:06,840 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: onGetSnapshot 12:24:06,840 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , registered listener akka://test/user/$$m 12:24:06,840 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 12:24:06,840 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: onGetSnapshot 12:24:06,840 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 12:24:06,840 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard1-config52, leaderId=member-1-shard-shard1-config52, leaderPayloadVersion=13] 12:24:06,840 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-shard1-config52 from Candidate to Leader 12:24:06,840 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 12:24:06,840 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 12:24:06,840 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from Candidate to Leader 12:24:06,840 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , registered listener akka://test/user/$$m 12:24:06,840 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard2-config52, leaderId=member-1-shard-shard2-config52, leaderPayloadVersion=13] 12:24:06,840 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-shard2-config52 from Candidate to Leader 12:24:06,840 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config52: All Shards are ready - data store config52 is ready 12:24:06,840 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard1-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@47e1a1fc, electionTerm=1, electionVotedFor=member-1-shard-shard1-config52, ServerConfigPayload=null]] 12:24:06,840 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard2-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@340d1ea, electionTerm=1, electionVotedFor=member-1-shard-shard2-config52, ServerConfigPayload=null]] 12:24:06,840 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: All shard snapshots received 12:24:06,851 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config52: onAddShardReplica: AddShardReplica[ShardName=astronauts] 12:24:06,852 PM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config52: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-609#-670219854 for shard member-1-shard-astronauts-config52 12:24:06,853 PM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-astronauts-config52, persistent : true 12:24:06,854 PM [test-akka.actor.default-dispatcher-4] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-astronauts-config52/shard-member-1:snapshot-read#1469875535] 12:24:06,854 PM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-astronauts-config52 with journal batch size 1 12:24:06,854 PM [test-akka.actor.default-dispatcher-17] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-astronauts-config52/member-1-shard-astronauts-config52-notifier#482182105 created and ready for shard:member-1-shard-astronauts-config52 12:24:06,854 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: Received AddServerReply{status=OK, leaderHint=} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-609#-670219854 12:24:06,854 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: Leader shard successfully added the replica shard astronauts 12:24:06,855 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-astronauts-config52 12:24:06,855 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: persisting the shard list [shard2, astronauts, shard1] 12:24:06,855 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManager - shard-manager-config52 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 12:24:06,855 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManager - shard-manager-config52: Successfully deleted prior snapshots 12:24:06,858 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 12:24:06,858 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-astronauts-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:24:06,858 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [astronauts] 12:24:06,858 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,858 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config52 is initialized 12:24:06,858 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 12:24:06,858 PM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from null to Follower 12:24:06,859 PM [test-shard-dispatcher-18] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: RaftPolicy used with prev.config org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy, RaftPolicy used with newConfig null 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: Re-initializing to Follower with previous leaderId null 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-astronauts-config52 (Follower): Closing 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52 (Follower): Received TimeoutNow - switching to Candidate 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52 (Candidate): Election: Candidate has following voting peers: [] 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: Set currentTerm=1, votedFor=member-1-shard-astronauts-config52 12:24:06,859 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-astronauts-config52 (Candidate): Starting new election term 1 12:24:06,859 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-astronauts-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-astronauts-config52 (Follower): Closing 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 12:24:06,859 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 12:24:06,859 PM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from Follower to Candidate 12:24:06,859 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : astronauts 12:24:06,860 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 12:24:06,860 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: onGetSnapshot 12:24:06,860 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Expecting 3 shard snapshot replies 12:24:06,860 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 12:24:06,860 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52 (Candidate): Received ElectionTimeout 12:24:06,860 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52 (Leader): Election: Leader has following peers: [] 12:24:06,860 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 12:24:06,860 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 12:24:06,860 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: true, hasLeader : true 12:24:06,860 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: onGetSnapshot 12:24:06,860 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 12:24:06,860 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: became leader with frontend state for [] 12:24:06,861 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 12:24:06,861 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard2-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@4da45220, electionTerm=1, electionVotedFor=member-1-shard-shard2-config52, ServerConfigPayload=null]] 12:24:06,861 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onGetSnapshot 12:24:06,861 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 12:24:06,861 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 12:24:06,861 PM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from Candidate to Leader 12:24:06,861 PM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , registered listener akka://test/user/$$m 12:24:06,861 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-astronauts-config52, leaderId=member-1-shard-astronauts-config52, leaderPayloadVersion=13] 12:24:06,861 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-astronauts-config52 from Candidate to Leader 12:24:06,861 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config52: All Shards are ready - data store config52 is ready 12:24:06,861 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard1-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@639542ea, electionTerm=1, electionVotedFor=member-1-shard-shard1-config52, ServerConfigPayload=null]] 12:24:06,862 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-astronauts-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@40998cdc, electionTerm=1, electionVotedFor=member-1-shard-astronauts-config52, ServerConfigPayload=null]] 12:24:06,862 PM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: All shard snapshots received 12:24:06,862 PM [main] [INFO] ShardManagerTest - testGetSnapshot ending 12:24:06,963 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$m] 12:24:06,974 PM [test-shard-dispatcher-20] [INFO] Shard - Stopping Shard member-1-shard-shard1-config52 12:24:06,974 PM [test-shard-dispatcher-20] [DEBUG] MessageSlicer - member-1-shard-shard1-config52 (Leader)_slicer-id-221: Closing 12:24:06,974 PM [test-shard-dispatcher-18] [INFO] Shard - Stopping Shard member-1-shard-shard2-config52 12:24:06,974 PM [test-shard-dispatcher-18] [DEBUG] MessageSlicer - member-1-shard-shard2-config52 (Leader)_slicer-id-222: Closing 12:24:06,974 PM [test-shard-dispatcher-25] [INFO] Shard - Stopping Shard member-1-shard-astronauts-config52 12:24:06,975 PM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config52: Aborting 0 pending queued transactions 12:24:06,975 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: Aborting 0 pending queued transactions 12:24:06,975 PM [test-shard-dispatcher-25] [DEBUG] MessageSlicer - member-1-shard-astronauts-config52 (Leader)_slicer-id-224: Closing 12:24:06,975 PM [test-shard-dispatcher-25] [DEBUG] Shard - member-1-shard-astronauts-config52: Aborting 0 pending queued transactions 12:24:06,981 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config52 12:24:06,985 PM [main] [INFO] ShardManagerTest - testWhenMultipleShardsPresentSyncStatusMustBeTrueForAllShards starting 12:24:06,989 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config53 12:24:06,990 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config53 12:24:06,990 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config53: Creating local shard: member-1-shard-default-config53 12:24:06,990 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config53: Creating local shard: member-1-shard-astronauts-config53 12:24:06,991 PM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-default-config53 from Follower to Leader 12:24:06,991 PM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-astronauts-config53 from Follower to Leader 12:24:06,991 PM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-astronauts-config53 from Leader to Follower 12:24:06,991 PM [main] [INFO] ShardManager - shard-manager-config53 Received follower initial sync status for member-1-shard-astronauts-config53 status sync done true 12:24:06,991 PM [main] [INFO] ShardManagerTest - testWhenMultipleShardsPresentSyncStatusMustBeTrueForAllShards ending 12:24:07,92 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$n] 12:24:07,103 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config53 12:24:07,105 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithFindPrimaryTimeout starting 12:24:07,107 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config54 12:24:07,108 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Recovery complete : shard-manager-config54 12:24:07,117 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:07,118 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config54: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://non-existent@127.0.0.1:5 12:24:07,118 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config54: All Shards are ready - data store config54 is ready 12:24:07,118 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config54: onAddShardReplica: AddShardReplica[ShardName=astronauts] 12:24:07,118 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config54: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=true] 12:24:07,118 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config54: findPrimary for astronauts forwarding to remote ShardManager akka://non-existent@127.0.0.1:5/user/shardmanager-config54, visitedAddresses: [akka://test@127.0.0.1:2550/user/shardmanager-config54] 12:24:07,137 PM [test-akka.actor.default-dispatcher-22] [WARN] Materializer - [outbound connection to [akka://non-existent@127.0.0.1:5], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1/:5,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused 12:24:07,138 PM [test-akka.actor.default-dispatcher-22] [WARN] Materializer - [outbound connection to [akka://non-existent@127.0.0.1:5], control stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1/:5,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused 12:24:07,332 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config54: Received failure from FindPrimary for shard astronauts akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://test/user/shardmanager-config54#2045359740]] after [200 ms]. Message of type [org.opendaylight.controller.cluster.datastore.messages.FindPrimary]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. 12:24:07,332 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithFindPrimaryTimeout ending 12:24:07,434 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/shardmanager-config54#2045359740] 12:24:07,445 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Stopping ShardManager shard-manager-config54 12:24:07,446 PM [main] [INFO] ShardManagerTest - testOnCreateShard starting 12:24:07,449 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Starting ShardManager shard-manager-config55 12:24:07,450 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config55 12:24:07,450 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 12:24:07,460 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config55: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@2b9aa3ff] 12:24:07,460 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config55 doCreateShard: shardId: member-1-shard-foo-config55, memberNames: [MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], peerAddresses: {member-6-shard-foo-config55=null, member-5-shard-foo-config55=null}, isActiveMember: true 12:24:07,461 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config55: findLocalShard : foo 12:24:07,461 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config55: Scheduling 300000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config55, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 12:24:07,463 PM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-foo-config55, persistent : false 12:24:07,464 PM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$C/member-1-shard-foo-config55/member-1-shard-foo-config55-notifier#660593907 created and ready for shard:member-1-shard-foo-config55 12:24:07,464 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$C/member-1-shard-foo-config55/shard-member-1:snapshot-read#1650463349] 12:24:07,466 PM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-foo-config55 with journal batch size 1 12:24:07,472 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-foo-config55: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 12:24:07,472 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-foo-config55: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:24:07,473 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config55: onLeadershipChange, isLeader: false, hasLeader : false 12:24:07,473 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-foo-config55: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config55 is no longer the leader 12:24:07,473 PM [test-akka.actor.default-dispatcher-16] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config55 , received role change from null to Follower 12:24:07,473 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config55: Initializing shard [foo] 12:24:07,473 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardInformation - Shard member-1-shard-foo-config55 is initialized 12:24:07,473 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardInformation - Shard member-1-shard-foo-config55 is initialized - notifying 1 OnShardInitialized callbacks 12:24:07,473 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config55 , registered listener akka://test/user/$C 12:24:07,473 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config55: Received role changed for member-1-shard-foo-config55 from null to Follower 12:24:07,473 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config55: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@2b9aa3ff] 12:24:07,473 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config55: Shard foo already exists 12:24:07,473 PM [main] [INFO] ShardManagerTest - testOnCreateShard ending 12:24:07,575 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$C#117401626] 12:24:07,586 PM [test-shard-dispatcher-18] [INFO] Shard - Stopping Shard member-1-shard-foo-config55 12:24:07,586 PM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-foo-config55 (Follower): Closing 12:24:07,586 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-foo-config55: Aborting 0 pending queued transactions 12:24:07,587 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config55 12:24:07,588 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Stopping ShardManager shard-manager-config1 12:24:07,588 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config35 12:24:07,588 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config11 12:24:07,588 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - Stopping ShardManager shard-manager-config43 12:24:07,588 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Stopping ShardManager shard-manager-config21 12:24:07,588 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config46 12:24:07,589 PM [test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:24:07,589 PM [test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [WARN] [03/13/2024 12:24:07.596] [test-akka.actor.internal-dispatcher-6] [Cluster(akka://test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config