09:28:56,383 AM [test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:28:56,392 AM [test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://test@127.0.0.1:2565] with UID [4591984651387965756] 09:28:56,393 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Starting up, Akka version [2.6.21] ... 09:28:56,398 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Registered cluster JMX MBean [akka:type=Cluster] 09:28:56,398 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Started up successfully 09:28:56,398 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:28:56,398 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:28:57,245 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$a#-1921824378] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:28:57,254 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config1 09:28:57,311 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config1 09:28:57,311 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Creating local shard: member-1-shard-default-config1 09:28:57,324 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:28:57,324 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config1 09:28:57,324 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Initializing shard [default] 09:28:57,324 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config1 is initialized 09:28:57,325 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config1: Received role changed for member-1-shard-default-config1 from null to Follower 09:28:57,325 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: onChangeShardServersVotingStatus: ChangeShardMembersVotingStatus [shardName=default, meberVotingStatusMap={member-2=true}] 09:28:57,329 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: findLocalShard : default 09:28:57,331 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Sending ChangeServersVotingStatus message ChangeServersVotingStatus [serverVotingStatusMap={member-2-shard-default-config1=true}, serversVisited=[]] to local shard akka://test/user/member-1-shard-default-config1 09:28:57,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config1: Received ServerChangeReply{status=NO_LEADER} from local shard akka://test/user/member-1-shard-default-config1 09:28:57,333 AM [test-akka.actor.default-dispatcher-11] [WARN] ShardManager - shard-manager-config1: ChangeServersVotingStatus failed for shard default with status NO_LEADER 09:28:57,471 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config1#496308359] 09:28:57,484 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForShardLeader starting 09:28:57,488 AM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$b#630312916] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:28:57,489 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config2 09:28:57,493 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config2 09:28:57,493 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config2: Creating local shard: member-1-shard-default-config2 09:28:57,500 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:28:57,500 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config2 09:28:57,503 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config2: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:28:57,508 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config2: Scheduling 10000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config2, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 09:28:57,654 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config2: Initializing shard [default] 09:28:57,654 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized 09:28:57,654 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized - notifying 1 OnShardInitialized callbacks 09:28:57,801 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config2: Received role changed for member-1-shard-default-config2 from Candidate to Leader 09:28:57,801 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized - notifying 1 OnShardInitialized callbacks 09:28:57,954 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config2: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config2, leaderId=member-1-shard-default-config2, leaderPayloadVersion=12] 09:28:57,954 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is ready - notifying 1 OnShardInitialized callbacks 09:28:57,960 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config2: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#1847421790, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$UzfQ6MyN@1e0bf505] 09:28:57,961 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config2: All Shards are ready - data store config2 is ready 09:28:58,161 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForShardLeader ending 09:28:58,263 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$b#630312916] 09:28:58,274 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config2 09:28:58,286 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config3 09:28:58,287 AM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/$$a] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:28:58,289 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config3 09:28:58,289 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config3: Creating local shard: member-1-shard-default-config3 09:28:58,388 AM [main] [INFO] ShardManager - shard-manager-config3: Received role changed for member-1-shard-default-config3 from Follower to Candidate 09:28:58,388 AM [main] [INFO] ShardManager - shard-manager-config3 Received follower initial sync status for member-1-shard-default-config3 status sync done true 09:28:58,490 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$a] 09:28:58,501 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config3 09:28:58,517 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:28:58,530 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [3968899006220894477] 09:28:58,530 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 09:28:58,537 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:28:58,537 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 09:28:58,543 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:28:58,543 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:28:58,569 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:28:58,569 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 09:28:58,569 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 09:28:58,574 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config4] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:28:58,587 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config4 09:28:58,591 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 09:28:58,619 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config4 09:28:58,619 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-1-shard-default-config4 09:28:58,619 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-1-shard-astronauts-config4 09:28:58,619 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 09:28:58,619 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 09:28:58,620 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 09:28:58,635 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-4591867275301371343] 09:28:58,635 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 09:28:58,649 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:28:58,649 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 09:28:58,652 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:28:58,652 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:28:58,669 AM [main] [ERROR] ShardManagerTest - Mock Shard Leader Actor : TestActor[akka://cluster-test/user/member-2-shard-default-config4] 09:28:58,673 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config4 09:28:58,673 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config4] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:28:58,674 AM [main] [ERROR] ShardManagerTest - Forwarding actor : Actor[akka://cluster-test/user/shardmanager-config4/member-2-shard-default-config4#902496389] 09:28:58,685 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:28:58,685 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config4 09:28:58,685 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 09:28:58,685 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 to akka://cluster-test/user/member-1-shard-default-config 09:28:58,686 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: updated peer member-1-shard-default-config4 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 on shard member-1-shard-default-config4 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#-644964294] 09:28:58,686 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-astronauts-config4 09:28:58,686 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 09:28:58,686 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 to akka://cluster-test/user/member-1-shard-default-config 09:28:58,686 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: updated peer member-1-shard-astronauts-config4 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 on shard member-1-shard-astronauts-config4 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#-644964294] 09:28:58,686 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: Initializing shard [default] 09:28:58,686 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config4 is initialized 09:28:58,686 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config4: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config4, leaderId=member-2-shard-default-config4, leaderPayloadVersion=11] 09:28:58,686 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config4: Received role changed for member-1-shard-default-config4 from Candidate to Follower 09:28:58,707 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - Recovery complete : shard-manager-config4 09:28:58,707 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-2-shard-default-config4 09:28:58,708 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:28:58,708 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config4 09:28:58,708 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config4: Initializing shard [default] 09:28:58,708 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardInformation - Shard member-2-shard-default-config4 is initialized 09:28:58,708 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config4: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config4, leaderId=member-2-shard-default-config4, leaderPayloadVersion=11] 09:28:58,709 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config4: Received role changed for member-2-shard-default-config4 from Candidate to Leader 09:28:58,709 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 09:28:59,73 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 09:28:59,170 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 09:28:59,173 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 09:28:59,174 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 09:28:59,174 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 to akka://cluster-test/user/member-2-shard-default-config4 09:28:59,174 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 09:28:59,564 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 09:28:59,565 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:28:59,565 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 09:28:59,565 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 to akka://cluster-test/user/member-1-shard-default-config 09:28:59,565 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-astronauts-config4 09:28:59,565 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-astronauts-config4 to akka://cluster-test/user/member-1-shard-default-config 09:28:59,566 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config4: onRemoveShardReplica: RemoveShardReplica [shardName=default, memberName=MemberName{name=member-1}] 09:28:59,568 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config4: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:28:59,568 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config4: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4, primaryVersion=11] 09:28:59,569 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: Sending RemoveServer message to peer akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 for shard member-1-shard-default-config4 09:28:59,589 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: Received RemoveServerReply{status=OK, leaderHint=member-2-shard-default-config4} for shard default 09:28:59,589 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: Leader shard successfully removed the replica shard default 09:28:59,675 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:28:59,675 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 09:28:59,675 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 to akka://cluster-test/user/member-2-shard-default-config4 09:28:59,675 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 09:28:59,717 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config4 09:28:59,718 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:28:59,718 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 09:28:59,744 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:28:59,744 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:28:59,755 AM [cluster-test-akka.actor.default-dispatcher-16] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2558], message stream] Upstream failed, cause: StreamTcpException: The connection closed with error: Connection reset 09:28:59,755 AM [cluster-test-akka.actor.default-dispatcher-16] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2558], control stream] Upstream failed, cause: StreamTcpException: The connection closed with error: Connection reset 09:28:59,800 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config4 09:28:59,802 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:28:59,802 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [WARN] [04/25/2024 09:29:00.828] [cluster-test-akka.actor.internal-dispatcher-4] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:00,848 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$c#49749192] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:00,848 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config5 09:29:00,851 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config5 09:29:00,851 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config5: Creating local shard: member-1-shard-default-config5 09:29:00,863 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:00,863 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config5 09:29:00,863 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config5: Initializing shard [default] 09:29:00,863 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config5 is initialized 09:29:01,17 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$c#49749192] 09:29:01,29 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config5 09:29:01,33 AM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/$$b] to Actor[akka://test/user/termination-monitor] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:01,34 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config6 09:29:01,35 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config6 09:29:01,36 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config6: Creating local shard: member-1-shard-default-config6 09:29:01,36 AM [main] [INFO] ShardManager - shard-manager-config6: Received role changed for member-1-shard-default-config6 from null to Follower 09:29:01,47 AM [main] [INFO] ShardManager - shard-manager-config6: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 09:29:01,47 AM [main] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config6 with address akka://test/user/shardmanager-config6/member-2-shard-default-config6 09:29:01,47 AM [main] [INFO] ShardManager - shard-manager-config6: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config6, leaderId=member-2-shard-default-config6, leaderPayloadVersion=12] 09:29:01,47 AM [main] [INFO] ShardManager - shard-manager-config6: All Shards are ready - data store config6 is ready 09:29:01,149 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$b] 09:29:01,159 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config6 09:29:01,165 AM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$d#2134792853] to Actor[akka://test/user/termination-monitor] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:01,166 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config7 09:29:01,167 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config7 09:29:01,176 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config7: onRemoveShardReplica: RemoveShardReplica [shardName=model-inventory, memberName=MemberName{name=member-1}] 09:29:01,177 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config7: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=model-inventory, waitUntilReady=true] 09:29:01,177 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config7: No shard found for model-inventory 09:29:01,177 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config7: Failed to find leader for shard model-inventory: received response: {} org.opendaylight.controller.cluster.datastore.exceptions.PrimaryNotFoundException: No primary shard found for model-inventory. at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.findPrimary(ShardManager.java:1060) at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.handleCommand(ShardManager.java:208) at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManagerTest$TestShardManager.handleCommand(ShardManagerTest.java:2198) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) at scala.PartialFunction.applyOrElse(PartialFunction.scala:214) at scala.PartialFunction.applyOrElse$(PartialFunction.scala:213) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:537) at akka.actor.Actor.aroundReceive$(Actor.scala:535) at akka.persistence.AbstractPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:295) at akka.persistence.Eventsourced$$anon$4.stateReceive(Eventsourced.scala:917) at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:245) at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:244) at akka.persistence.AbstractPersistentActor.aroundReceive(PersistentActor.scala:295) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579) at akka.actor.ActorCell.invoke(ActorCell.scala:547) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) at akka.dispatch.Mailbox.run(Mailbox.scala:231) at akka.dispatch.Mailbox.exec(Mailbox.scala:243) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) 09:29:01,279 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$d#2134792853] 09:29:01,290 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config7 09:29:01,292 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingLocalReplicaLeader starting 09:29:01,294 AM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$e#1428327451] to Actor[akka://test/user/termination-monitor] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:01,295 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config8 09:29:01,296 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config8 09:29:01,296 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config8: Creating local shard: member-1-shard-default-config8 09:29:01,305 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:01,305 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config8 09:29:01,306 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config8: Initializing shard [default] 09:29:01,306 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config8 is initialized 09:29:01,306 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config8: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config8, leaderId=member-1-shard-default-config8, leaderPayloadVersion=12] 09:29:01,306 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config8: Received role changed for member-1-shard-default-config8 from Candidate to Leader 09:29:01,306 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config8: All Shards are ready - data store config8 is ready 09:29:01,306 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config8: onAddShardReplica: AddShardReplica[ShardName=default] 09:29:01,308 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config8: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:01,308 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config8: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#-1755278110, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$UzfQ6MyN@5422a1a9] 09:29:01,308 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config8: Local shard default already exists 09:29:01,309 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config8: findLocalShard : default 09:29:01,309 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingLocalReplicaLeader ending 09:29:01,411 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$e#1428327451] 09:29:01,422 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config8 09:29:01,424 AM [main] [INFO] ShardManagerTest - testOnReceiveFindLocalShardWaitForShardInitialized starting 09:29:01,427 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$f#28971775] to Actor[akka://test/user/termination-monitor] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:01,427 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config9 09:29:01,431 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config9 09:29:01,431 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config9: Creating local shard: member-1-shard-default-config9 09:29:01,441 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:01,441 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config9 09:29:01,442 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config9: findLocalShard : default 09:29:01,442 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config9: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config9, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 09:29:01,442 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config9: Initializing shard [default] 09:29:01,442 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config9 is initialized 09:29:01,442 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config9 is initialized - notifying 1 OnShardInitialized callbacks 09:29:01,446 AM [main] [INFO] ShardManagerTest - testOnReceiveFindLocalShardWaitForShardInitialized starting 09:29:01,547 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$f#28971775] 09:29:01,558 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config9 09:29:01,561 AM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberUnreachableAndLeadershipChange starting 09:29:01,570 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:29:01,584 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [-7738465129762277883] 09:29:01,584 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 09:29:01,591 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 09:29:01,591 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 09:29:01,591 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:29:01,591 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:29:01,602 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 09:29:01,602 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 09:29:01,602 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 09:29:01,605 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config10] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:01,607 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config10 09:29:01,612 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config10 09:29:01,612 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Creating local shard: member-1-shard-default-config10 09:29:01,612 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 09:29:01,612 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 09:29:01,614 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:29:01,625 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [7464549617627258416] 09:29:01,626 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 09:29:01,628 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:01,628 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 09:29:01,645 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:29:01,645 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:29:01,652 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config10] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:01,653 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config10 09:29:01,659 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config10 09:29:01,659 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Creating local shard: member-2-shard-default-config10 09:29:01,665 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:01,665 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config10 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 to akka://cluster-test/user/member-1-shard-default-config 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: updated peer member-1-shard-default-config10 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 on shard member-1-shard-default-config10 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#1816334738] 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Initializing shard [default] 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config10 is initialized 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config10, leaderId=member-2-shard-default-config10, leaderPayloadVersion=12] 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received role changed for member-1-shard-default-config10 from Candidate to Follower 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config10 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config10: Initializing shard [default] 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-2-shard-default-config10 is initialized 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config10, leaderId=member-2-shard-default-config10, leaderPayloadVersion=12] 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config10: Received role changed for member-2-shard-default-config10 from Candidate to Leader 09:29:01,666 AM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 09:29:01,744 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 09:29:01,750 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 09:29:01,750 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 09:29:01,750 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 to akka://cluster-test/user/member-2-shard-default-config 09:29:01,750 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 09:29:01,750 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 09:29:02,599 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 09:29:02,600 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:29:02,600 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 09:29:02,600 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 to akka://cluster-test/user/member-1-shard-default-config 09:29:02,600 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 09:29:02,600 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:02,600 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config10: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10, primaryVersion=12] 09:29:02,601 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 09:29:02,601 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config10 as unavailable. 09:29:02,604 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:02,604 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config10, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config10, activeMember=true] 09:29:02,651 AM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:29:02,651 AM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 09:29:02,651 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 to akka://cluster-test/user/member-2-shard-default-config 09:29:02,651 AM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 09:29:03,817 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Received ShardNotInitializedTimeout message for shard default 09:29:03,817 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Returning NoShardLeaderException for shard default 09:29:03,818 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config10, leaderId=member-1-shard-default-config10, leaderPayloadVersion=12] 09:29:03,818 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 09:29:03,818 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received role changed for member-1-shard-default-config10 from Follower to Leader 09:29:03,818 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 09:29:03,818 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:03,818 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config10: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/member-1-shard-default-config#1816334738, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$UzfQ6MyN@1625dba2] 09:29:03,818 AM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberUnreachableAndLeadershipChange ending 09:29:03,921 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config10 09:29:03,923 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:03,923 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 09:29:03,932 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:29:03,932 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:29:03,935 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 09:29:03,950 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config10 09:29:03,951 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:03,951 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 09:29:04,966 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [04/25/2024 09:29:04.968] [cluster-test-akka.actor.internal-dispatcher-2] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:04,988 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/shardmanager-config11] to Actor[akka://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:04,988 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config11 09:29:04,990 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config11 09:29:05,10 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:05,10 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config11: onAddShardReplica: AddShardReplica[ShardName=astronauts] 09:29:05,12 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config11: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-533#226750673 for shard member-1-shard-astronauts-config11 09:29:05,23 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config11: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-2}] 09:29:05,23 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config11: A shard replica operation for astronauts is already in progress 09:29:05,126 AM [main] [INFO] ShardManagerTest - testPerShardDatastoreContext starting 09:29:05,135 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config12 09:29:05,137 AM [test-akka.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$g#-620146014] to Actor[akka://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:05,138 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config12 09:29:05,138 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config12: Creating local shard: member-1-shard-default-config12 09:29:05,139 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config12: Creating local shard: member-1-shard-topology-config12 09:29:05,148 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:05,148 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config12 09:29:05,148 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-topology-config12 09:29:05,149 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-default-config12 09:29:05,149 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-topology-config12 09:29:05,199 AM [main] [INFO] ShardManagerTest - testPerShardDatastoreContext ending 09:29:05,300 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/default1#1991743472] 09:29:05,311 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/topology2#1447069016] 09:29:05,311 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$g#-620146014] 09:29:05,312 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config12 09:29:05,313 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForLocalLeaderShard starting 09:29:05,316 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config13 09:29:05,319 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config13 09:29:05,319 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config13: Creating local shard: member-1-shard-default-config13 09:29:05,328 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:05,328 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config13 09:29:05,328 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config13: Initializing shard [default] 09:29:05,328 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config13 is initialized 09:29:05,329 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config13: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config13, leaderId=member-1-shard-default-config13, leaderPayloadVersion=12] 09:29:05,379 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config13: Received role changed for member-1-shard-default-config13 from Candidate to Leader 09:29:05,379 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config13: All Shards are ready - data store config13 is ready 09:29:05,379 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config13: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 09:29:05,380 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config13: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#1268414012, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$UzfQ6MyN@240d263c] 09:29:05,380 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForLocalLeaderShard ending 09:29:05,482 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$h#232258254] 09:29:05,492 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config13 09:29:05,495 AM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberWithNameContainedInLeaderIdUnreachable starting 09:29:05,504 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 09:29:05,516 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2562] with UID [2658175869844985663] 09:29:05,516 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Starting up, Akka version [2.6.21] ... 09:29:05,516 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Registered cluster JMX MBean [akka:type=Cluster] 09:29:05,516 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Started up successfully 09:29:05,516 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:29:05,517 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:29:05,536 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Starting ShardManager shard-manager-config14 09:29:05,536 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Node [akka://cluster-test@127.0.0.1:2562] is JOINING itself (with roles [member-256, dc-default], version [0.0.0]) and forming new cluster 09:29:05,536 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - is the new leader among reachable nodes (more leaders may exist) 09:29:05,536 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Leader is moving node [akka://cluster-test@127.0.0.1:2562] to [Up] 09:29:05,536 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config14] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:05,539 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:29:05,542 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Recovery complete : shard-manager-config14 09:29:05,542 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config14: Creating local shard: member-256-shard-default-config14 09:29:05,542 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-256}, address: akka://cluster-test@127.0.0.1:2562 09:29:05,542 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 09:29:05,550 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [6839866287934792337] 09:29:05,551 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 09:29:05,553 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:05,553 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 09:29:05,559 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:29:05,559 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:29:05,572 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config14] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:05,575 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config14 09:29:05,577 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:05,577 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - Creating Shard member-256-shard-default-config14 09:29:05,578 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 09:29:05,578 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 to akka://cluster-test/user/member-256-shard-default-config 09:29:05,578 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config14: updated peer member-256-shard-default-config14 on member MemberName{name=member-256} with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 on shard member-256-shard-default-config14 whose actor address is Actor[akka://cluster-test/user/member-256-shard-default-config#-720704144] 09:29:05,578 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config14: Initializing shard [default] 09:29:05,578 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Shard member-256-shard-default-config14 is initialized 09:29:05,578 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config14: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-256-shard-default-config14, leaderId=member-256-shard-default-config14, leaderPayloadVersion=12] 09:29:05,578 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config14: Received role changed for member-256-shard-default-config14 from Candidate to Leader 09:29:05,578 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 09:29:05,579 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config14 09:29:05,579 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config14: Creating local shard: member-2-shard-default-config14 09:29:05,579 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:05,579 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config14 09:29:05,579 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config14: Initializing shard [default] 09:29:05,579 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-2-shard-default-config14 is initialized 09:29:05,579 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config14: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config14, leaderId=member-256-shard-default-config14, leaderPayloadVersion=12] 09:29:05,580 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config14: Received role changed for member-2-shard-default-config14 from Candidate to Follower 09:29:05,604 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 09:29:05,608 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2562] 09:29:05,609 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-256}, address: akka://cluster-test@127.0.0.1:2562 09:29:05,609 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 09:29:05,609 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 to akka://cluster-test/user/member-2-shard-default-config 09:29:05,609 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 09:29:06,533 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 09:29:06,533 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:29:06,533 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 09:29:06,533 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 to akka://cluster-test/user/member-256-shard-default-config 09:29:06,533 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 09:29:06,534 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config14: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:06,534 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config14: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2562/user/member-256-shard-default-config#-720704144, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$UzfQ6MyN@798d86eb] 09:29:06,535 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 09:29:06,535 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config14: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:06,536 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config14: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2562/user/member-256-shard-default-config#-720704144, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$UzfQ6MyN@798d86eb] 09:29:06,537 AM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberWithNameContainedInLeaderIdUnreachable ending 09:29:06,582 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:29:06,582 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 09:29:06,582 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 to akka://cluster-test/user/member-2-shard-default-config 09:29:06,582 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 09:29:06,640 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - Stopping ShardManager shard-manager-config14 09:29:06,640 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:06,640 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 09:29:06,649 AM [cluster-test-akka.actor.default-dispatcher-6] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:29:06,649 AM [cluster-test-akka.actor.default-dispatcher-6] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:29:06,672 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shard-manager-config14 09:29:06,674 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:06,674 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 09:29:07,592 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - is the new leader among reachable nodes (more leaders may exist) 09:29:07,699 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [04/25/2024 09:29:07.702] [cluster-test-akka.actor.internal-dispatcher-2] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:07,717 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config15 09:29:07,720 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config15 09:29:07,720 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config15: Creating local shard: member-1-shard-default-config15 09:29:07,721 AM [main] [INFO] ShardManager - shard-manager-config15: Received role changed for member-1-shard-default-config15 from Follower to Leader 09:29:07,822 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$c] 09:29:07,833 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config15 09:29:07,839 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config16 09:29:07,840 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config16 09:29:07,840 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config16: Creating local shard: member-1-shard-default-config16 09:29:07,841 AM [main] [INFO] ShardManager - shard-manager-config16: Received role changed for member-1-shard-default-config16 from Candidate to Leader 09:29:07,841 AM [main] [INFO] ShardManager - shard-manager-config16: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config16, leaderId=member-1-shard-default-config16, leaderPayloadVersion=12] 09:29:07,841 AM [main] [INFO] ShardManager - shard-manager-config16: All Shards are ready - data store config16 is ready 09:29:07,943 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$d] 09:29:07,954 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config16 09:29:07,956 AM [main] [INFO] ShardManagerTest - testOnCreateShardWithLocalMemberNotInShardConfig starting 09:29:07,958 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config17 09:29:07,962 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config17 09:29:07,962 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:07,975 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config17: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@5f75145b] 09:29:07,976 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config17 doCreateShard: shardId: member-1-shard-foo-config17, memberNames: [MemberName{name=member-5}, MemberName{name=member-6}], peerAddresses: {}, isActiveMember: false 09:29:07,981 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config17: findLocalShard : foo 09:29:07,981 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config17: Scheduling 60000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config17, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=false] 09:29:07,994 AM [test-shard-dispatcher-15] [INFO] Shard - Shard created : member-1-shard-foo-config17, persistent : true 09:29:08,10 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$i/member-1-shard-foo-config17/member-1-shard-foo-config17-notifier#-195195798 created and ready for shard:member-1-shard-foo-config17 09:29:08,16 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$i/member-1-shard-foo-config17/shard-member-1:snapshot-read#1617026186] 09:29:08,20 AM [test-shard-dispatcher-15] [INFO] Shard - Starting recovery for member-1-shard-foo-config17 with journal batch size 1 09:29:08,31 AM [test-shard-dispatcher-15] [DEBUG] Shard - member-1-shard-foo-config17: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 09:29:08,33 AM [test-shard-dispatcher-15] [INFO] Shard - member-1-shard-foo-config17: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:29:08,34 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config17: Initializing shard [foo] 09:29:08,34 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-foo-config17 is initialized 09:29:08,34 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-foo-config17 is initialized - notifying 1 OnShardInitialized callbacks 09:29:08,34 AM [main] [INFO] ShardManagerTest - testOnCreateShardWithLocalMemberNotInShardConfig ending 09:29:08,40 AM [test-shard-dispatcher-15] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config17: onLeadershipChange, isLeader: false, hasLeader : false 09:29:08,40 AM [test-shard-dispatcher-15] [DEBUG] Shard - member-1-shard-foo-config17: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config17 is no longer the leader 09:29:08,41 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config17 , received role change from null to Follower 09:29:08,44 AM [test-shard-dispatcher-15] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 09:29:08,46 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config17 , registered listener akka://test/user/$i 09:29:08,46 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config17: Received role changed for member-1-shard-foo-config17 from null to Follower 09:29:08,136 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$i#-1516286546] 09:29:08,147 AM [test-shard-dispatcher-15] [INFO] Shard - Stopping Shard member-1-shard-foo-config17 09:29:08,147 AM [test-shard-dispatcher-15] [DEBUG] MessageAssembler - member-1-shard-foo-config17 (Follower): Closing 09:29:08,148 AM [test-shard-dispatcher-15] [DEBUG] Shard - member-1-shard-foo-config17: Aborting 0 pending queued transactions 09:29:08,148 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config17 09:29:08,151 AM [main] [INFO] ShardManagerTest - testServerRemovedShardActorNotRunning starting 09:29:08,154 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config18 09:29:08,155 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config18 09:29:08,155 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config18: Creating local shard: member-1-shard-default-config18 09:29:08,155 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config18: Creating local shard: member-1-shard-people-config18 09:29:08,166 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: findLocalShard : people 09:29:08,166 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: Returning NotInitializedException for shard people 09:29:08,167 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: findLocalShard : default 09:29:08,167 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: Returning NotInitializedException for shard default 09:29:08,168 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18 : Local Shard replica for shard default has been removed 09:29:08,169 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: persisting the shard list [people] 09:29:08,169 AM [main] [INFO] ShardManagerTest - testServerRemovedShardActorNotRunning ending 09:29:08,177 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 09:29:08,180 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: Successfully deleted prior snapshots 09:29:08,270 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$e] 09:29:08,280 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config18 09:29:08,285 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config19 09:29:08,286 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config19 09:29:08,286 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config19: Creating local shard: member-1-shard-default-config19 09:29:08,295 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:08,295 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config19 09:29:08,295 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config19: Initializing shard [default] 09:29:08,295 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config19 is initialized 09:29:08,296 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config19: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 09:29:08,296 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config19: Returning NoShardLeaderException for shard default 09:29:08,397 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$j#-1014945548] 09:29:08,408 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config19 09:29:08,410 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForFollowerShardWithNoInitialLeaderId starting 09:29:08,412 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config20 09:29:08,413 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config20 09:29:08,413 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config20: Creating local shard: member-1-shard-default-config20 09:29:08,422 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:08,422 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config20 09:29:08,422 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: Initializing shard [default] 09:29:08,422 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config20 is initialized 09:29:08,423 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config20: Received role changed for member-1-shard-default-config20 from Candidate to Follower 09:29:08,423 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 09:29:08,423 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: Returning NoShardLeaderException for shard default 09:29:08,423 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config20: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config20, leaderId=member-1-shard-default-config20, leaderPayloadVersion=12] 09:29:08,423 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config20: All Shards are ready - data store config20 is ready 09:29:08,423 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 09:29:08,423 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#-278784293, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$UzfQ6MyN@567f5757] 09:29:08,423 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForFollowerShardWithNoInitialLeaderId starting 09:29:08,525 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$k#321254114] 09:29:08,536 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config20 09:29:08,540 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config21 09:29:08,542 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config21 09:29:08,561 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:08,561 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config21: onAddShardReplica: AddShardReplica[ShardName=astronauts] 09:29:08,562 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config21: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-551#1213002303 for shard member-1-shard-astronauts-config21 09:29:08,573 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config21: onAddShardReplica: AddShardReplica[ShardName=astronauts] 09:29:08,573 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config21: A shard replica operation for astronauts is already in progress 09:29:08,677 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithCandidateShard starting 09:29:08,680 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config22 09:29:08,682 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config22 09:29:08,682 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config22: Creating local shard: member-1-shard-default-config22 09:29:08,691 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:08,691 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config22 09:29:08,691 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: Initializing shard [default] 09:29:08,691 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config22 is initialized 09:29:08,691 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config22: Received role changed for member-1-shard-default-config22 from null to Candidate 09:29:08,691 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:08,691 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config22, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Candidate, leaderId=null, activeMember=true] 09:29:09,904 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: Received ShardNotInitializedTimeout message for shard default 09:29:09,904 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: Returning NoShardLeaderException for shard default 09:29:09,905 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithCandidateShard ending 09:29:10,8 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$l#1540209862] 09:29:10,19 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config22 09:29:10,21 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithAddServerReplyFailure starting 09:29:10,24 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config23 09:29:10,25 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config23 09:29:10,144 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:10,154 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config23: onAddShardReplica: AddShardReplica[ShardName=astronauts] 09:29:10,155 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config23: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-556#1528483770 for shard member-1-shard-astronauts-config23 09:29:10,156 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config23: Received AddServerReply{status=TIMEOUT} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-556#1528483770 09:29:10,156 AM [test-akka.actor.default-dispatcher-5] [WARN] ShardManager - shard-manager-config23: Leader failed to add shard replica astronauts with status TIMEOUT 09:29:10,157 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config23: findLocalShard : astronauts 09:29:10,158 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config23: Local shard astronauts not found - shards present: [] 09:29:10,159 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config23: onAddShardReplica: AddShardReplica[ShardName=astronauts] 09:29:10,160 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config23: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-556#1528483770 for shard member-1-shard-astronauts-config23 09:29:10,160 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config23: Received AddServerReply{status=NO_LEADER} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-556#1528483770 09:29:10,160 AM [test-akka.actor.default-dispatcher-5] [WARN] ShardManager - shard-manager-config23: Leader failed to add shard replica astronauts with status NO_LEADER 09:29:10,160 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithAddServerReplyFailure ending 09:29:10,262 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-astronauts-config#2146895989] 09:29:10,272 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/shardmanager-config23] 09:29:10,273 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config23 09:29:10,279 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config24 09:29:10,281 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config24 09:29:10,281 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config24: Creating local shard: member-1-shard-default-config24 09:29:10,282 AM [main] [INFO] ShardManager - shard-manager-config24: Received role changed for member-1-shard-default-config24 from Candidate to Follower 09:29:10,282 AM [main] [INFO] ShardManager - shard-manager-config24 Received follower initial sync status for member-1-shard-default-config24 status sync done true 09:29:10,282 AM [main] [INFO] ShardManager - shard-manager-config24 Received follower initial sync status for member-1-shard-default-config24 status sync done false 09:29:10,383 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$f] 09:29:10,394 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config24 09:29:10,398 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config25 09:29:10,401 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config25 09:29:10,401 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config25: Creating local shard: member-1-shard-default-config25 09:29:10,401 AM [main] [INFO] ShardManager - shard-manager-config25: Received role changed for unknown from Candidate to Leader 09:29:10,503 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$g] 09:29:10,514 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config25 09:29:10,516 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithUninitializedShard starting 09:29:10,518 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config26 09:29:10,520 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config26 09:29:10,520 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config26: Creating local shard: member-1-shard-default-config26 09:29:10,529 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:10,529 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config26 09:29:10,529 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:10,529 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config26, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 09:29:11,144 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: Received ShardNotInitializedTimeout message for shard default 09:29:11,144 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: Returning NotInitializedException for shard default 09:29:11,145 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: Initializing shard [default] 09:29:11,145 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config26 is initialized 09:29:11,345 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithUninitializedShard ending 09:29:11,446 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$m#1221015836] 09:29:11,458 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config26 09:29:11,463 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config27 09:29:11,465 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config27 09:29:11,465 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config27: Creating local shard: member-1-shard-default-config27 09:29:11,474 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:11,474 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config27 09:29:11,474 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config27: Initializing shard [default] 09:29:11,474 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config27 is initialized 09:29:11,474 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config27: findLocalShard : default 09:29:11,575 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$n#-1872576569] 09:29:11,586 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config27 09:29:11,587 AM [main] [INFO] ShardManagerTest - testRegisterForShardLeaderChanges starting 09:29:11,590 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config28 09:29:11,591 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config28 09:29:11,592 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config28: Creating local shard: member-1-shard-default-config28 09:29:11,600 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:11,600 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config28 09:29:11,600 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config28: Initializing shard [default] 09:29:11,600 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config28 is initialized 09:29:11,600 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config28: onRegisterForShardAvailabilityChanges: org.opendaylight.controller.cluster.datastore.shardmanager.RegisterForShardAvailabilityChanges@1e2e55fd 09:29:11,601 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=12] 09:29:11,631 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=12] 09:29:12,132 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=null, leaderPayloadVersion=12] 09:29:12,143 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-2-shard-default-config28, leaderPayloadVersion=12] 09:29:12,154 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=12] 09:29:12,655 AM [main] [INFO] ShardManagerTest - testRegisterForShardLeaderChanges ending 09:29:12,756 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$o#560190748] 09:29:12,767 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config28 09:29:12,772 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config29 09:29:12,773 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config29 09:29:12,773 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config29: Creating local shard: member-1-shard-default-config29 09:29:12,975 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$h] 09:29:12,986 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config29 09:29:12,990 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config30 09:29:12,991 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config30 09:29:12,991 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config30: Creating local shard: member-1-shard-default-config30 09:29:13,1 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:13,1 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config30 09:29:13,1 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config30: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=non-existent, waitUntilReady=false] 09:29:13,1 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config30: No shard found for non-existent 09:29:13,102 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$p#-1714111036] 09:29:13,113 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config30 09:29:13,118 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config31 09:29:13,119 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config31 09:29:13,128 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config31: onAddShardReplica: AddShardReplica[ShardName=model-inventory] 09:29:13,128 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config31: No module configuration exists for shard model-inventory 09:29:13,230 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$q#9894935] 09:29:13,241 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config31 09:29:13,242 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingReplicaInRemoteShardLeader starting 09:29:13,244 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config32 09:29:13,245 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config32 09:29:13,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Creating local shard: member-1-shard-default-config32 09:29:13,356 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:13,356 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config32 09:29:13,356 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Initializing shard [default] 09:29:13,356 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config32 is initialized 09:29:13,356 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config32: Received MemberUp: memberName: MemberName{name=leader-member}, address: akka://test 09:29:13,356 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer leader-member-shard-default-config32 with address akka://test/user/shardmanager-config32/leader-member-shard-default-config32 09:29:13,356 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer leader-member-shard-default-config32 with address akka://test/user/shardmanager-config32/leader-member-shard-default-config32 to akka://test/user/member-1-shard-default-config 09:29:13,356 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config32: Received role changed for member-1-shard-default-config32 from Candidate to Follower 09:29:13,356 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config32: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config32, leaderId=leader-member-shard-default-config32, leaderPayloadVersion=12] 09:29:13,357 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config32: All Shards are ready - data store config32 is ready 09:29:13,357 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 09:29:13,357 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:13,357 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=12] 09:29:13,357 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 09:29:13,357 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Received AddServerReply{status=ALREADY_EXISTS} for shard default from leader akka://test/user/shardmanager-config32/leader-member-shard-default-config32 09:29:13,358 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Local shard default already exists 09:29:13,408 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: findLocalShard : default 09:29:13,408 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 09:29:13,408 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:13,409 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=12] 09:29:13,409 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 09:29:13,409 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config32: Received AddServerReply{status=ALREADY_EXISTS} for shard default from leader akka://test/user/shardmanager-config32/leader-member-shard-default-config32 09:29:13,409 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config32: Local shard default already exists 09:29:13,410 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-default-config32 09:29:13,410 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 09:29:13,410 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:13,411 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=12] 09:29:13,411 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 09:29:13,525 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config32: AddServer request to akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for default failed akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://test/), Path(/user/shardmanager-config32/leader-member-shard-default-config32)]] after [100 ms]. Message of type [org.opendaylight.controller.cluster.raft.messages.AddServer]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. 09:29:13,526 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config32: findLocalShard : default 09:29:13,526 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingReplicaInRemoteShardLeader ending 09:29:13,628 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/shardmanager-config32] 09:29:13,639 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config32 09:29:13,641 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForRemoteShard starting 09:29:13,647 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:29:13,658 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [6791273241545838072] 09:29:13,658 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 09:29:13,660 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 09:29:13,660 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 09:29:13,668 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:29:13,668 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:29:13,675 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 09:29:13,675 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 09:29:13,675 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 09:29:13,676 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config33] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:13,677 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config33 09:29:13,683 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:29:13,684 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Recovery complete : shard-manager-config33 09:29:13,684 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-1-shard-default-config33 09:29:13,684 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 09:29:13,684 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 09:29:13,690 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [5699875656231134536] 09:29:13,690 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 09:29:13,692 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:13,692 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 09:29:13,692 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:29:13,692 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:29:13,706 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config33] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:13,707 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config33 09:29:13,709 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config33 09:29:13,709 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-2-shard-default-config33 09:29:13,710 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-2-shard-astronauts-config33 09:29:13,714 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:13,714 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config33 09:29:13,714 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-2-shard-astronauts-config33 09:29:13,714 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config33: Initializing shard [astronauts] 09:29:13,714 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-2-shard-astronauts-config33 is initialized 09:29:13,714 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config33: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-astronauts-config33, leaderId=member-2-shard-astronauts-config33, leaderPayloadVersion=11] 09:29:13,714 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config33: Received role changed for member-2-shard-astronauts-config33 from Candidate to Leader 09:29:13,715 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:13,715 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config33 09:29:13,715 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 09:29:13,715 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 to akka://test/user/member-1-shard-default-config 09:29:13,715 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: updated peer member-1-shard-default-config33 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 on shard member-1-shard-default-config33 whose actor address is Actor[akka://test/user/member-1-shard-default-config#-157801924] 09:29:13,766 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 09:29:13,776 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 09:29:13,776 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 09:29:13,776 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 09:29:13,776 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 09:29:13,776 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-astronauts-config33 09:29:13,776 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-astronauts-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 09:29:14,688 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 09:29:14,688 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:29:14,688 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 09:29:14,688 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 to akka://test/user/member-1-shard-default-config 09:29:14,689 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config33: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=false] 09:29:14,689 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config33: findPrimary for astronauts forwarding to remote ShardManager akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33, visitedAddresses: [akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33] 09:29:14,696 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config33: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.RemoteFindPrimary [shardName=astronauts, waitUntilReady=false] 09:29:14,696 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config33: Found primary for astronauts: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config#1783082962, primaryVersion=11] 09:29:14,699 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForRemoteShard ending 09:29:14,716 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:29:14,716 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 09:29:14,716 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 09:29:14,716 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-astronauts-config33 09:29:14,716 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-astronauts-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 09:29:14,801 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config33 09:29:14,802 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:14,802 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 09:29:14,810 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:29:14,810 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:29:14,827 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config33 09:29:14,827 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:14,827 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 09:29:15,848 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [04/25/2024 09:29:15.850] [cluster-test-akka.actor.internal-dispatcher-8] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:15,861 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config34 09:29:15,864 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config34 09:29:15,864 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config34: Creating local shard: member-1-shard-default-config34 09:29:15,871 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config34: findLocalShard : default 09:29:15,871 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config34: Returning NotInitializedException for shard default 09:29:15,973 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$r#1904844449] 09:29:15,984 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config34 09:29:15,988 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config35 09:29:15,989 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config35 09:29:15,989 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config35: Creating local shard: member-1-shard-default-config35 09:29:15,997 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:15,997 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config35 09:29:15,997 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config35: Initializing shard [default] 09:29:15,997 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config35 is initialized 09:29:15,998 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config35: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config35, leaderId=member-1-shard-default-config35, leaderPayloadVersion=12] 09:29:15,998 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config35: Received role changed for member-1-shard-default-config35 from Candidate to Leader 09:29:15,998 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config35: All Shards are ready - data store config35 is ready 09:29:15,998 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config35: onChangeShardServersVotingStatus: ChangeShardMembersVotingStatus [shardName=default, meberVotingStatusMap={member-2=true}] 09:29:15,998 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config35: findLocalShard : default 09:29:15,998 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config35: Sending ChangeServersVotingStatus message ChangeServersVotingStatus [serverVotingStatusMap={member-2-shard-default-config35=true}, serversVisited=[]] to local shard akka://test/user/member-1-shard-default-config35 09:29:15,998 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config35: Received ServerChangeReply{status=OK} from local shard akka://test/user/member-1-shard-default-config35 09:29:15,998 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config35: ChangeServersVotingStatus succeeded for shard default 09:29:16,150 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config35#362497997] 09:29:16,162 AM [main] [INFO] ShardManagerTest - testShutDown starting 09:29:16,168 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config36 09:29:16,169 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config36 09:29:16,169 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: Creating local shard: member-1-shard-shard2-config36 09:29:16,169 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: Creating local shard: member-1-shard-shard1-config36 09:29:16,179 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:16,179 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config36 09:29:16,179 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config36 09:29:16,180 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config36: Initializing shard [shard1] 09:29:16,180 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config36 is initialized 09:29:16,180 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config36: Initializing shard [shard2] 09:29:16,180 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config36 is initialized 09:29:16,180 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config36: Issuing gracefulStop to shard member-1-shard-shard2-config36 09:29:16,181 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config36: Issuing gracefulStop to shard member-1-shard-shard1-config36 09:29:16,181 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Shutting down ShardManager shard-manager-config36 - waiting on 2 shards 09:29:16,732 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-shard1-config36#-206395182] 09:29:16,732 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-shard2-config36#471047726] 09:29:16,733 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config36: All shards shutdown - sending PoisonPill to self 09:29:16,733 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config36 09:29:16,734 AM [main] [INFO] ShardManagerTest - testShutDown ending 09:29:16,835 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$t#-341789865] 09:29:16,847 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShardBeforeMemberUp starting 09:29:16,850 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config37 09:29:16,856 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config37 09:29:16,856 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config37: Creating local shard: member-1-shard-default-config37 09:29:16,861 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:16,861 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config37 09:29:16,861 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config37: Initializing shard [default] 09:29:16,861 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config37 is initialized 09:29:16,861 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config37: Received role changed for member-1-shard-default-config37 from Candidate to Follower 09:29:16,861 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerTest$TestShardManager - Received unhandled message LeaderStateChanged [memberId=member-1-shard-default-config37, leaderId=member-2-shard-default-config37, leaderPayloadVersion=12] 09:29:16,866 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config37: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 09:29:16,866 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config37: Returning NoShardLeaderException for shard default 09:29:16,867 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShardBeforeMemberUp ending 09:29:16,969 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$u#351782281] 09:29:16,980 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config37 09:29:16,982 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithNoRoleShard starting 09:29:16,985 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config38 09:29:16,986 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config38 09:29:16,986 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config38: Creating local shard: member-1-shard-default-config38 09:29:16,995 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:16,995 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config38 09:29:16,995 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: Initializing shard [default] 09:29:16,995 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config38 is initialized 09:29:16,995 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:16,996 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config38, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 09:29:18,215 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: Received ShardNotInitializedTimeout message for shard default 09:29:18,215 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: Returning NoShardLeaderException for shard default 09:29:18,215 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithNoRoleShard ending 09:29:18,317 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$v#1235867322] 09:29:18,328 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config38 09:29:18,330 AM [main] [INFO] ShardManagerTest - testShardPersistenceWithRestoredData starting 09:29:18,334 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config39 09:29:18,335 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config39: onSnapshotOffer: ShardManagerSnapshot [ShardList = [default, astronauts] ] 09:29:18,335 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config39: adding shard astronauts 09:29:18,335 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config39: removing shard people 09:29:18,336 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config39 09:29:18,337 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Creating local shard: member-1-shard-default-config39 09:29:18,337 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Creating local shard: member-1-shard-astronauts-config39 09:29:18,444 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : people 09:29:18,444 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Local shard people not found - shards present: [default, astronauts] 09:29:18,444 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : default 09:29:18,444 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Returning NotInitializedException for shard default 09:29:18,444 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : astronauts 09:29:18,444 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Returning NotInitializedException for shard astronauts 09:29:18,445 AM [main] [INFO] ShardManagerTest - testShardPersistenceWithRestoredData ending 09:29:18,546 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$i] 09:29:18,557 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config39 09:29:18,559 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShard starting 09:29:18,562 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config40 09:29:18,563 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config40 09:29:18,563 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config40: Creating local shard: member-1-shard-default-config40 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config40 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config40: Initializing shard [default] 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config40 is initialized 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config40: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config40 with address akka://test/user/shardmanager-config40/member-2-shard-default-config40 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config40 with address akka://test/user/shardmanager-config40/member-2-shard-default-config40 to akka://test/user/member-1-shard-default-config 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config40: Received role changed for member-1-shard-default-config40 from Candidate to Follower 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config40: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config40, leaderId=member-2-shard-default-config40, leaderPayloadVersion=11] 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config40: All Shards are ready - data store config40 is ready 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config40: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 09:29:18,573 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config40: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config40/member-2-shard-default-config40, primaryVersion=11] 09:29:18,573 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShard ending 09:29:18,675 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$w#1479827303] 09:29:18,685 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config40 09:29:18,689 AM [main] [INFO] ShardManagerTest - testShardAvailabilityOnChangeOfMemberReachability starting 09:29:18,699 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:29:18,706 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [-6142790816253306382] 09:29:18,707 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 09:29:18,708 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 09:29:18,708 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 09:29:18,717 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:29:18,717 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:29:18,727 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 09:29:18,730 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config41 09:29:18,730 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config41] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:18,730 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 09:29:18,736 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 09:29:18,743 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 09:29:18,747 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config41 09:29:18,747 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Creating local shard: member-1-shard-default-config41 09:29:18,748 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 09:29:18,748 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 09:29:18,753 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-414563832407848613] 09:29:18,753 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 09:29:18,755 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:18,755 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 09:29:18,755 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:29:18,755 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:29:18,775 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config41] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:18,778 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:18,778 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config41 09:29:18,778 AM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 09:29:18,778 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 09:29:18,778 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config41: updated peer member-1-shard-default-config41 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 on shard member-1-shard-default-config41 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#1808370486] 09:29:18,778 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config41: Initializing shard [default] 09:29:18,778 AM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is initialized 09:29:18,778 AM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config41: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config41, leaderId=member-2-shard-default-config41, leaderPayloadVersion=12] 09:29:18,778 AM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config41: Received role changed for member-1-shard-default-config41 from Candidate to Follower 09:29:18,780 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config41 09:29:18,791 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Recovery complete : shard-manager-config41 09:29:18,791 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config41: Creating local shard: member-2-shard-default-config41 09:29:18,792 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:18,792 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config41 09:29:18,792 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config41: Initializing shard [default] 09:29:18,792 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardInformation - Shard member-2-shard-default-config41 is initialized 09:29:18,792 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config41: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config41, leaderId=member-2-shard-default-config41, leaderPayloadVersion=12] 09:29:18,792 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config41: Received role changed for member-2-shard-default-config41 from Candidate to Leader 09:29:18,792 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 09:29:18,823 AM [cluster-test-akka.actor.default-dispatcher-15] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 09:29:18,827 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 09:29:18,828 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 09:29:18,828 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 09:29:18,828 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 to akka://cluster-test/user/member-2-shard-default-config 09:29:18,828 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 09:29:19,735 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 09:29:19,736 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:29:19,736 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 09:29:19,736 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 09:29:19,736 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 09:29:19,736 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:19,736 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=12] 09:29:19,736 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 09:29:19,737 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as unavailable. 09:29:19,738 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config41: Received MemberRemoved: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 09:29:19,738 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:19,739 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config41, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config41, activeMember=true] 09:29:19,777 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:29:19,778 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 09:29:19,778 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-2-shard-default-config 09:29:19,778 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 09:29:20,953 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Received ShardNotInitializedTimeout message for shard default 09:29:20,953 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Returning NoShardLeaderException for shard default 09:29:20,954 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Received ReachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 09:29:20,954 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 09:29:20,954 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 09:29:20,954 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as available. 09:29:20,954 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:20,954 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=12] 09:29:20,955 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 09:29:20,955 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 09:29:20,955 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 09:29:20,955 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 09:29:20,955 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 09:29:20,956 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as unavailable. 09:29:20,956 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:20,956 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config41, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config41, activeMember=true] 09:29:20,956 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Received ReachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 09:29:20,956 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 09:29:20,956 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 09:29:20,956 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is initialized - notifying 1 OnShardInitialized callbacks 09:29:20,956 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as available. 09:29:20,956 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is ready - notifying 1 OnShardInitialized callbacks 09:29:20,956 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=12] 09:29:20,956 AM [main] [INFO] ShardManagerTest - testShardAvailabilityOnChangeOfMemberReachability ending 09:29:21,59 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config41 09:29:21,60 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:21,60 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 09:29:21,67 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:29:21,67 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:29:21,69 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 09:29:21,86 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Stopping ShardManager shard-manager-config41 09:29:21,87 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:21,87 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 09:29:22,102 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [04/25/2024 09:29:22.104] [cluster-test-akka.actor.internal-dispatcher-2] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:22,111 AM [main] [INFO] ShardManagerTest - testAddShardReplica starting 09:29:22,118 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:29:22,128 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [-500431319836254119] 09:29:22,128 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 09:29:22,137 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 09:29:22,137 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 09:29:22,140 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:29:22,140 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:29:22,145 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 09:29:22,145 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 09:29:22,145 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 09:29:22,147 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shardmanager-config42 09:29:22,147 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config42] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:22,149 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 09:29:22,153 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shardmanager-config42 09:29:22,153 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-1-shard-default-config42 09:29:22,154 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 09:29:22,154 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 09:29:22,160 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-5934950521416078298] 09:29:22,161 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 09:29:22,162 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:22,162 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 09:29:22,169 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 09:29:22,169 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 09:29:22,186 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:22,186 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config42 09:29:22,186 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 09:29:22,186 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 to akka://cluster-test/user/member-1-shard-default-config 09:29:22,186 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shardmanager-config42: updated peer member-1-shard-default-config42 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 on shard member-1-shard-default-config42 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#1140808576] 09:29:22,188 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config42] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 09:29:22,192 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shardmanager-config42 09:29:22,195 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shardmanager-config42 09:29:22,195 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-2-shard-default-config42 09:29:22,195 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-2-shard-astronauts-config42 09:29:22,196 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:22,196 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config42 09:29:22,196 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-2-shard-astronauts-config42 09:29:22,196 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: Initializing shard [astronauts] 09:29:22,196 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-2-shard-astronauts-config42 is initialized 09:29:22,196 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shardmanager-config42: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-astronauts-config42, leaderId=member-2-shard-astronauts-config42, leaderPayloadVersion=11] 09:29:22,196 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shardmanager-config42: Received role changed for member-2-shard-astronauts-config42 from Candidate to Leader 09:29:22,233 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 09:29:22,235 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 09:29:22,236 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 09:29:22,236 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 09:29:22,236 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 09:29:22,236 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-astronauts-config42 09:29:22,236 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-astronauts-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 09:29:23,157 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 09:29:23,157 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:29:23,157 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 09:29:23,157 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 to akka://cluster-test/user/member-1-shard-default-config 09:29:23,161 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: onAddShardReplica: AddShardReplica[ShardName=astronauts] 09:29:23,162 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=true] 09:29:23,162 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: findPrimary for astronauts forwarding to remote ShardManager akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42, visitedAddresses: [akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42] 09:29:23,165 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.RemoteFindPrimary [shardName=astronauts, waitUntilReady=true] 09:29:23,165 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42: Found primary for astronauts: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config, primaryVersion=11] 09:29:23,171 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: Sending AddServer message to peer akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config for shard member-1-shard-astronauts-config42 09:29:23,178 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: Received AddServerReply{status=OK, leaderHint=member-2-shard-astronauts-config42} for shard astronauts from leader akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config 09:29:23,178 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: Leader shard successfully added the replica shard astronauts 09:29:23,178 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-astronauts-config42 09:29:23,178 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: persisting the shard list [default, astronauts] 09:29:23,179 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 09:29:23,180 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: Successfully deleted prior snapshots 09:29:23,182 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 09:29:23,182 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 09:29:23,182 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 09:29:23,182 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-astronauts-config42 09:29:23,182 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-astronauts-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 09:29:23,213 AM [main] [INFO] ShardManagerTest - testAddShardReplica ending 09:29:23,315 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shardmanager-config42 09:29:23,316 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:23,316 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [WARN] [04/25/2024 09:29:23.330] [cluster-test-akka.remote.default-remote-dispatcher-6] [akka.stream.Log(akka://cluster-test/system/Materializers/StreamSupervisor-604)] [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted [WARN] [04/25/2024 09:29:23.331] [cluster-test-akka.remote.default-remote-dispatcher-7] [akka.stream.Log(akka://cluster-test/system/Materializers/StreamSupervisor-604)] [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 09:29:23,345 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shardmanager-config42 09:29:23,348 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:23,348 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 09:29:24,358 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [04/25/2024 09:29:24.361] [cluster-test-akka.actor.internal-dispatcher-9] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 09:29:24,373 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config43 09:29:24,377 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config43 09:29:24,377 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config43: Creating local shard: member-1-shard-default-config43 09:29:24,383 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:24,383 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config43 09:29:24,383 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Initializing shard [default] 09:29:24,383 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config43 is initialized 09:29:24,383 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config43: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config43, leaderId=member-1-shard-default-config43, leaderPayloadVersion=12] 09:29:24,383 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config43: Received role changed for member-1-shard-default-config43 from Candidate to Leader 09:29:24,383 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config43: All Shards are ready - data store config43 is ready 09:29:24,383 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: onRemoveShardReplica: RemoveShardReplica [shardName=default, memberName=MemberName{name=member-1}] 09:29:24,383 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:24,383 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config43#1292626974, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$UzfQ6MyN@5025832c] 09:29:24,384 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config43: Sending RemoveServer message to peer akka://test@127.0.0.1:2565/user/member-1-shard-default-config43#1292626974 for shard member-1-shard-default-config43 09:29:24,384 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Received RemoveServerReply{status=OK} for shard default 09:29:24,384 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Leader shard successfully removed the replica shard default 09:29:24,535 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config43#1292626974] 09:29:24,548 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithIsolatedLeaderShard starting 09:29:24,550 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config44 09:29:24,551 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config44 09:29:24,551 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: Creating local shard: member-1-shard-default-config44 09:29:24,560 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:24,560 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config44 09:29:24,561 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config44: Initializing shard [default] 09:29:24,561 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config44 is initialized 09:29:24,561 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config44: Received role changed for member-1-shard-default-config44 from null to IsolatedLeader 09:29:24,561 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config44: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 09:29:24,561 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config44: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config44, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=IsolatedLeader, leaderId=null, activeMember=true] 09:29:25,774 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config44: Received ShardNotInitializedTimeout message for shard default 09:29:25,774 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config44: Returning NoShardLeaderException for shard default 09:29:25,775 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithIsolatedLeaderShard ending 09:29:25,877 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$y#1913148335] 09:29:25,888 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config44 09:29:25,892 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config45 09:29:25,893 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config45 09:29:25,893 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config45: Creating local shard: member-1-shard-default-config45 09:29:25,902 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:25,902 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config45 09:29:25,902 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config45: findLocalShard : non-existent 09:29:25,902 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config45: Local shard non-existent not found - shards present: [default] 09:29:26,4 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$z#582045461] 09:29:26,15 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config45 09:29:26,19 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config46 09:29:26,20 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config46 09:29:26,40 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:26,40 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config46: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-2}] 09:29:26,40 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config46: Sending RemoveServer message to peer akka://test@127.0.0.1:2565/system/testProbe-598#-1124191061 for shard member-2-shard-astronauts-config46 09:29:26,51 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config46: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-3}] 09:29:26,51 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config46: A shard replica operation for astronauts is already in progress 09:29:26,156 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config47 09:29:26,157 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config47 09:29:26,157 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config47: Creating local shard: member-1-shard-default-config47 09:29:26,166 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config47: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 09:29:26,166 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config47: Returning NotInitializedException for shard default 09:29:26,268 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$A#-629398957] 09:29:26,279 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config47 09:29:26,281 AM [main] [INFO] ShardManagerTest - testServerRemovedShardActorRunning starting 09:29:26,284 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config48 09:29:26,286 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config48 09:29:26,286 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config48: Creating local shard: member-1-shard-default-config48 09:29:26,286 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config48: Creating local shard: member-1-shard-people-config48 09:29:26,297 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:26,297 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config48 09:29:26,297 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-people-config48 09:29:26,297 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config48: Initializing shard [default] 09:29:26,297 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config48 is initialized 09:29:26,297 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config48: findLocalShard : people 09:29:26,298 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config48: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-people-config48, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 09:29:26,300 AM [test-shard-dispatcher-16] [INFO] Shard - Shard created : member-1-shard-people-config48, persistent : true 09:29:26,303 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$j/member-1-shard-people-config48/member-1-shard-people-config48-notifier#994783498 created and ready for shard:member-1-shard-people-config48 09:29:26,303 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$j/member-1-shard-people-config48/shard-member-1:snapshot-read#-108493798] 09:29:26,303 AM [test-shard-dispatcher-16] [INFO] Shard - Starting recovery for member-1-shard-people-config48 with journal batch size 1 09:29:26,305 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-people-config48: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 09:29:26,305 AM [test-shard-dispatcher-16] [INFO] Shard - member-1-shard-people-config48: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:29:26,305 AM [test-shard-dispatcher-16] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-people-config48: onLeadershipChange, isLeader: false, hasLeader : false 09:29:26,305 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-people-config48: onStateChanged: Closing all transaction chains because shard member-1-shard-people-config48 is no longer the leader 09:29:26,305 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config48: Initializing shard [people] 09:29:26,306 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-people-config48 is initialized 09:29:26,306 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-people-config48 is initialized - notifying 1 OnShardInitialized callbacks 09:29:26,306 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-people-config48 , received role change from null to Follower 09:29:26,306 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config48: findLocalShard : default 09:29:26,306 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-people-config48 , registered listener akka://test/user/$$j 09:29:26,306 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config48 : Sending Shutdown to Shard actor Actor[akka://test/user/member-1-shard-default-config48#-1076309237] with 10000 ms timeout 09:29:26,309 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config48 : Local Shard replica for shard default has been removed 09:29:26,309 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config48: persisting the shard list [people] 09:29:26,309 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config48: Received role changed for member-1-shard-people-config48 from null to Follower 09:29:26,309 AM [main] [INFO] ShardManagerTest - testServerRemovedShardActorRunning ending 09:29:26,310 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config48 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 09:29:26,310 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config48: Successfully deleted prior snapshots 09:29:26,410 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config48#-1076309237] 09:29:26,421 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$j] 09:29:26,422 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48 : Successfully shut down Shard actor Actor[akka://test/user/member-1-shard-default-config48#-1076309237] 09:29:26,422 AM [test-shard-dispatcher-16] [INFO] Shard - Stopping Shard member-1-shard-people-config48 09:29:26,422 AM [test-shard-dispatcher-16] [DEBUG] MessageAssembler - member-1-shard-people-config48 (Follower): Closing 09:29:26,422 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-people-config48: Aborting 0 pending queued transactions 09:29:26,423 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config48 09:29:26,427 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config49 09:29:26,428 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config49 09:29:26,428 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config49: Creating local shard: member-1-shard-default-config49 09:29:26,528 AM [main] [INFO] ShardManager - shard-manager-config49: Received role changed for member-1-shard-default-config49 from null to Follower 09:29:26,528 AM [main] [INFO] ShardManager - shard-manager-config49: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config49, leaderId=member-2-shard-default-config49, leaderPayloadVersion=12] 09:29:26,539 AM [main] [INFO] ShardManager - shard-manager-config49: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 09:29:26,539 AM [main] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config49 with address akka://test/user/shardmanager-config49/member-2-shard-default-config49 09:29:26,539 AM [main] [INFO] ShardManager - shard-manager-config49: All Shards are ready - data store config49 is ready 09:29:26,641 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$k] 09:29:26,651 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config49 09:29:26,654 AM [main] [INFO] ShardManagerTest - testRestoreFromSnapshot starting 09:29:26,657 AM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Starting ShardManager shard-manager-config50 09:29:26,659 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config50 09:29:26,659 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Restoring from ShardManagerSnapshot: ShardManagerSnapshot [ShardList = [shard1, shard2, astronauts] ] 09:29:26,659 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: onSnapshotOffer: ShardManagerSnapshot [ShardList = [shard1, shard2, astronauts] ] 09:29:26,659 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: adding shard shard1 09:29:26,659 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: adding shard shard2 09:29:26,659 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: adding shard astronauts 09:29:26,659 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-shard2-config50 09:29:26,660 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-astronauts-config50 09:29:26,660 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-shard1-config50 09:29:26,757 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:26,757 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config50 09:29:26,757 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-1-shard-astronauts-config50 09:29:26,758 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config50 09:29:26,759 AM [test-shard-dispatcher-16] [INFO] Shard - Shard created : member-1-shard-shard2-config50, persistent : true 09:29:26,760 AM [test-shard-dispatcher-16] [INFO] Shard - Starting recovery for member-1-shard-shard2-config50 with journal batch size 1 09:29:26,761 AM [test-shard-dispatcher-16] [INFO] Shard - Shard created : member-1-shard-astronauts-config50, persistent : true 09:29:26,762 AM [test-shard-dispatcher-16] [INFO] Shard - Starting recovery for member-1-shard-astronauts-config50 with journal batch size 1 09:29:26,762 AM [test-shard-dispatcher-16] [INFO] Shard - Shard created : member-1-shard-shard1-config50, persistent : true 09:29:26,763 AM [test-shard-dispatcher-16] [INFO] Shard - Starting recovery for member-1-shard-shard1-config50 with journal batch size 1 09:29:26,763 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-shard2-config50/shard-member-1:snapshot-read#803382605] 09:29:26,763 AM [test-akka.actor.default-dispatcher-17] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-shard2-config50/member-1-shard-shard2-config50-notifier#-2033255476 created and ready for shard:member-1-shard-shard2-config50 09:29:26,763 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-astronauts-config50/shard-member-1:snapshot-read#-957320972] 09:29:26,763 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-shard1-config50/shard-member-1:snapshot-read#2047290684] 09:29:26,764 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-shard1-config50/member-1-shard-shard1-config50-notifier#1108003859 created and ready for shard:member-1-shard-shard1-config50 09:29:26,764 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-astronauts-config50/member-1-shard-astronauts-config50-notifier#-2050775670 created and ready for shard:member-1-shard-astronauts-config50 09:29:26,765 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 09:29:26,765 AM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard1-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:29:26,765 AM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: false, hasLeader : false 09:29:26,765 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config50 is no longer the leader 09:29:26,765 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [shard1] 09:29:26,765 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config50 is initialized 09:29:26,765 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from null to Follower 09:29:26,765 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-shard2-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 09:29:26,765 AM [test-shard-dispatcher-16] [INFO] Shard - member-1-shard-shard2-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:29:26,765 AM [test-shard-dispatcher-16] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: false, hasLeader : false 09:29:26,765 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-shard2-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config50 is no longer the leader 09:29:26,765 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [shard2] 09:29:26,765 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config50 is initialized 09:29:26,766 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from null to Follower 09:29:26,766 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-shard2-config50 (Follower): Received TimeoutNow - switching to Candidate 09:29:26,767 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : shard1 09:29:26,767 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : shard2 09:29:26,767 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : astronauts 09:29:26,767 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config50: Scheduling 3000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-astronauts-config50, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 09:29:26,768 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-astronauts-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 09:29:26,768 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-astronauts-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:29:26,768 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-shard2-config50 (Candidate): Election: Candidate has following voting peers: [] 09:29:26,768 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-shard2-config50: Set currentTerm=1, votedFor=member-1-shard-shard2-config50 09:29:26,768 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [astronauts] 09:29:26,768 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config50 is initialized 09:29:26,768 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config50 is initialized - notifying 1 OnShardInitialized callbacks 09:29:26,768 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: false, hasLeader : false 09:29:26,768 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-astronauts-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config50 is no longer the leader 09:29:26,769 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-astronauts-config50 (Follower): Received TimeoutNow - switching to Candidate 09:29:26,769 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-astronauts-config50 (Candidate): Election: Candidate has following voting peers: [] 09:29:26,769 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-astronauts-config50: Set currentTerm=1, votedFor=member-1-shard-astronauts-config50 09:29:26,769 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50 (Follower): Received TimeoutNow - switching to Candidate 09:29:26,769 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50 (Candidate): Election: Candidate has following voting peers: [] 09:29:26,769 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50: Set currentTerm=1, votedFor=member-1-shard-shard1-config50 09:29:26,769 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from null to Follower 09:29:26,770 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config50: onGetSnapshot 09:29:26,770 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Expecting 3 shard snapshot replies 09:29:26,771 AM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard1-config50 (Candidate): Starting new election term 1 09:29:26,771 AM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard1-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 09:29:26,771 AM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-shard1-config50 (Follower): Closing 09:29:26,771 AM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: false, hasLeader : false 09:29:26,771 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config50 is no longer the leader 09:29:26,771 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-astronauts-config50 (Candidate): Starting new election term 1 09:29:26,771 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-astronauts-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 09:29:26,771 AM [test-shard-dispatcher-20] [DEBUG] MessageAssembler - member-1-shard-astronauts-config50 (Follower): Closing 09:29:26,771 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: false, hasLeader : false 09:29:26,771 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-astronauts-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config50 is no longer the leader 09:29:26,771 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from Follower to Candidate 09:29:26,771 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from Follower to Candidate 09:29:26,775 AM [test-shard-dispatcher-16] [INFO] Shard - member-1-shard-shard2-config50 (Candidate): Starting new election term 1 09:29:26,775 AM [test-shard-dispatcher-16] [INFO] Shard - member-1-shard-shard2-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 09:29:26,775 AM [test-shard-dispatcher-16] [DEBUG] MessageAssembler - member-1-shard-shard2-config50 (Follower): Closing 09:29:26,775 AM [test-shard-dispatcher-16] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: false, hasLeader : false 09:29:26,775 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-shard2-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config50 is no longer the leader 09:29:26,778 AM [test-akka.actor.default-dispatcher-19] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from Follower to Candidate 09:29:26,781 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50 (Candidate): Received ElectionTimeout 09:29:26,781 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-astronauts-config50 (Candidate): Received ElectionTimeout 09:29:26,781 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50 (Candidate): Received ElectionTimeout 09:29:26,789 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50 (Leader): Election: Leader has following peers: [] 09:29:26,789 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-astronauts-config50 (Leader): Election: Leader has following peers: [] 09:29:26,789 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50 (Leader): Election: Leader has following peers: [] 09:29:26,789 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-shard2-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 09:29:26,789 AM [test-shard-dispatcher-16] [INFO] Shard - member-1-shard-astronauts-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 09:29:26,789 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-shard1-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 09:29:26,789 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: true, hasLeader : true 09:29:26,789 AM [test-shard-dispatcher-16] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: true, hasLeader : true 09:29:26,789 AM [test-shard-dispatcher-22] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: true, hasLeader : true 09:29:26,790 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50: became leader with frontend state for [] 09:29:26,790 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-astronauts-config50: became leader with frontend state for [] 09:29:26,790 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50: became leader with frontend state for [] 09:29:26,791 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from Candidate to Leader 09:29:26,791 AM [test-akka.actor.default-dispatcher-19] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from Candidate to Leader 09:29:26,791 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from Candidate to Leader 09:29:26,794 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50: onGetSnapshot 09:29:26,794 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50: onGetSnapshot 09:29:26,794 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , registered listener akka://test/user/$$l 09:29:26,794 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 09:29:26,794 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 09:29:26,794 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard2-config50, leaderId=member-1-shard-shard2-config50, leaderPayloadVersion=12] 09:29:26,794 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , registered listener akka://test/user/$$l 09:29:26,795 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-shard2-config50 from Candidate to Leader 09:29:26,795 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard1-config50, leaderId=member-1-shard-shard1-config50, leaderPayloadVersion=12] 09:29:26,795 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-shard1-config50 from Candidate to Leader 09:29:26,795 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-astronauts-config50: onGetSnapshot 09:29:26,795 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-astronauts-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 09:29:26,795 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , registered listener akka://test/user/$$l 09:29:26,795 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-astronauts-config50, leaderId=member-1-shard-astronauts-config50, leaderPayloadVersion=12] 09:29:26,795 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-astronauts-config50 from Candidate to Leader 09:29:26,795 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: All Shards are ready - data store config50 is ready 09:29:26,799 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 09:29:26,799 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-astronauts-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 09:29:26,800 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-astronauts-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@2aaf03d1, electionTerm=1, electionVotedFor=member-1-shard-astronauts-config50, ServerConfigPayload=null]] 09:29:26,801 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-shard2-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@3926af94, electionTerm=1, electionVotedFor=member-1-shard-shard2-config50, ServerConfigPayload=null]] 09:29:26,801 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 09:29:26,802 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-shard1-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@75d79f79, electionTerm=1, electionVotedFor=member-1-shard-shard1-config50, ServerConfigPayload=null]] 09:29:26,802 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: All shard snapshots received 09:29:26,803 AM [main] [INFO] ShardManagerTest - testRestoreFromSnapshot ending 09:29:26,905 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$l] 09:29:26,916 AM [test-shard-dispatcher-22] [INFO] Shard - Stopping Shard member-1-shard-shard2-config50 09:29:26,916 AM [test-shard-dispatcher-20] [INFO] Shard - Stopping Shard member-1-shard-astronauts-config50 09:29:26,916 AM [test-shard-dispatcher-22] [DEBUG] MessageSlicer - member-1-shard-shard2-config50 (Leader)_slicer-id-214: Closing 09:29:26,916 AM [test-shard-dispatcher-20] [DEBUG] MessageSlicer - member-1-shard-astronauts-config50 (Leader)_slicer-id-215: Closing 09:29:26,916 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config50: Aborting 0 pending queued transactions 09:29:26,916 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-astronauts-config50: Aborting 0 pending queued transactions 09:29:26,916 AM [test-shard-dispatcher-20] [INFO] Shard - Stopping Shard member-1-shard-shard1-config50 09:29:26,916 AM [test-shard-dispatcher-20] [DEBUG] MessageSlicer - member-1-shard-shard1-config50 (Leader)_slicer-id-216: Closing 09:29:26,916 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config50: Aborting 0 pending queued transactions 09:29:26,916 AM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Stopping ShardManager shard-manager-config50 09:29:26,918 AM [main] [INFO] ShardManagerTest - testOnCreateShardWithNoInitialSchemaContext starting 09:29:26,921 AM [test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Starting ShardManager shard-manager-config51 09:29:26,922 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - Recovery complete : shard-manager-config51 09:29:26,932 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config51: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@5942432] 09:29:26,932 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config51 doCreateShard: shardId: member-1-shard-foo-config51, memberNames: [MemberName{name=member-1}], peerAddresses: {}, isActiveMember: true 09:29:26,932 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:26,932 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-1-shard-foo-config51 09:29:26,933 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config51: findLocalShard : foo 09:29:26,933 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config51: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config51, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 09:29:26,933 AM [test-shard-dispatcher-20] [INFO] Shard - Shard created : member-1-shard-foo-config51, persistent : true 09:29:26,934 AM [test-akka.actor.default-dispatcher-17] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$B/member-1-shard-foo-config51/member-1-shard-foo-config51-notifier#531471284 created and ready for shard:member-1-shard-foo-config51 09:29:26,934 AM [test-akka.actor.default-dispatcher-17] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$B/member-1-shard-foo-config51/shard-member-1:snapshot-read#301401736] 09:29:26,934 AM [test-shard-dispatcher-20] [INFO] Shard - Starting recovery for member-1-shard-foo-config51 with journal batch size 1 09:29:26,935 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config51: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 09:29:26,935 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-foo-config51: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:29:26,935 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManager - shard-manager-config51: Initializing shard [foo] 09:29:26,935 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardInformation - Shard member-1-shard-foo-config51 is initialized 09:29:26,935 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardInformation - Shard member-1-shard-foo-config51 is initialized - notifying 1 OnShardInitialized callbacks 09:29:26,935 AM [main] [INFO] ShardManagerTest - testOnCreateShardWithNoInitialSchemaContext ending 09:29:26,935 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: false, hasLeader : false 09:29:26,935 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config51: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config51 is no longer the leader 09:29:26,936 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config51 (Follower): Received TimeoutNow - switching to Candidate 09:29:26,936 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config51 (Candidate): Election: Candidate has following voting peers: [] 09:29:26,936 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config51: Set currentTerm=1, votedFor=member-1-shard-foo-config51 09:29:26,936 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from null to Follower 09:29:26,936 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-foo-config51 (Candidate): Starting new election term 1 09:29:26,936 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-foo-config51 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 09:29:26,936 AM [test-shard-dispatcher-20] [DEBUG] MessageAssembler - member-1-shard-foo-config51 (Follower): Closing 09:29:26,936 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: false, hasLeader : false 09:29:26,936 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config51: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config51 is no longer the leader 09:29:26,936 AM [test-akka.actor.default-dispatcher-19] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from Follower to Candidate 09:29:26,937 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config51 (Candidate): Received ElectionTimeout 09:29:26,937 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config51 (Leader): Election: Leader has following peers: [] 09:29:26,937 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-foo-config51 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 09:29:26,937 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: true, hasLeader : true 09:29:26,937 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config51: became leader with frontend state for [] 09:29:26,937 AM [test-akka.actor.default-dispatcher-19] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from Candidate to Leader 09:29:26,937 AM [test-akka.actor.default-dispatcher-19] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , registered listener akka://test/user/$B 09:29:26,937 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - shard-manager-config51: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-foo-config51, leaderId=member-1-shard-foo-config51, leaderPayloadVersion=12] 09:29:26,937 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - shard-manager-config51: Received role changed for member-1-shard-foo-config51 from Candidate to Leader 09:29:26,937 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - shard-manager-config51: All Shards are ready - data store config51 is ready 09:29:27,37 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$B#1393523002] 09:29:27,48 AM [test-shard-dispatcher-20] [INFO] Shard - Stopping Shard member-1-shard-foo-config51 09:29:27,48 AM [test-shard-dispatcher-20] [DEBUG] MessageSlicer - member-1-shard-foo-config51 (Leader)_slicer-id-218: Closing 09:29:27,48 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config51: Aborting 0 pending queued transactions 09:29:27,49 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - Stopping ShardManager shard-manager-config51 09:29:27,50 AM [main] [INFO] ShardManagerTest - testGetSnapshot starting 09:29:27,53 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - Starting ShardManager shard-manager-config52 09:29:27,55 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - Recovery complete : shard-manager-config52 09:29:27,55 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManager - shard-manager-config52: Creating local shard: member-1-shard-shard2-config52 09:29:27,55 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManager - shard-manager-config52: Creating local shard: member-1-shard-shard1-config52 09:29:27,65 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 09:29:27,65 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:27,65 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config52 09:29:27,66 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config52 09:29:27,66 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : shard1 09:29:27,66 AM [test-shard-dispatcher-20] [INFO] Shard - Shard created : member-1-shard-shard2-config52, persistent : true 09:29:27,67 AM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-shard1-config52, persistent : true 09:29:27,67 AM [test-shard-dispatcher-20] [INFO] Shard - Starting recovery for member-1-shard-shard2-config52 with journal batch size 1 09:29:27,68 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-shard2-config52/member-1-shard-shard2-config52-notifier#-1423046117 created and ready for shard:member-1-shard-shard2-config52 09:29:27,68 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-shard2-config52/shard-member-1:snapshot-read#1489354323] 09:29:27,68 AM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-shard1-config52 with journal batch size 1 09:29:27,68 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-shard1-config52/member-1-shard-shard1-config52-notifier#-1386076448 created and ready for shard:member-1-shard-shard1-config52 09:29:27,69 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-shard1-config52/shard-member-1:snapshot-read#66377214] 09:29:27,70 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 09:29:27,70 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-shard2-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:29:27,70 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: false, hasLeader : false 09:29:27,70 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config52 is no longer the leader 09:29:27,70 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from null to Follower 09:29:27,70 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52 (Follower): Received TimeoutNow - switching to Candidate 09:29:27,70 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-shard1-config52, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 09:29:27,70 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [shard2] 09:29:27,70 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52 (Candidate): Election: Candidate has following voting peers: [] 09:29:27,70 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config52 is initialized 09:29:27,70 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: Set currentTerm=1, votedFor=member-1-shard-shard2-config52 09:29:27,70 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-shard2-config52 (Candidate): Starting new election term 1 09:29:27,70 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-shard2-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 09:29:27,70 AM [test-shard-dispatcher-20] [DEBUG] MessageAssembler - member-1-shard-shard2-config52 (Follower): Closing 09:29:27,70 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: false, hasLeader : false 09:29:27,70 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config52 is no longer the leader 09:29:27,71 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from Follower to Candidate 09:29:27,71 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 09:29:27,71 AM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard1-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:29:27,71 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [shard1] 09:29:27,71 AM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: false, hasLeader : false 09:29:27,71 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config52 is initialized 09:29:27,71 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config52 is no longer the leader 09:29:27,71 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config52 is initialized - notifying 1 OnShardInitialized callbacks 09:29:27,72 AM [test-akka.actor.default-dispatcher-24] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from null to Follower 09:29:27,72 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52 (Follower): Received TimeoutNow - switching to Candidate 09:29:27,72 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52 (Candidate): Election: Candidate has following voting peers: [] 09:29:27,72 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: Set currentTerm=1, votedFor=member-1-shard-shard1-config52 09:29:27,72 AM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard1-config52 (Candidate): Starting new election term 1 09:29:27,72 AM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard1-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 09:29:27,72 AM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-shard1-config52 (Follower): Closing 09:29:27,72 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52 (Candidate): Received ElectionTimeout 09:29:27,72 AM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: false, hasLeader : false 09:29:27,72 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config52 is no longer the leader 09:29:27,72 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52 (Leader): Election: Leader has following peers: [] 09:29:27,72 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-shard2-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 09:29:27,72 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: true, hasLeader : true 09:29:27,72 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from Follower to Candidate 09:29:27,72 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: became leader with frontend state for [] 09:29:27,72 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : shard2 09:29:27,72 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from Candidate to Leader 09:29:27,72 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , registered listener akka://test/user/$$m 09:29:27,72 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard2-config52, leaderId=member-1-shard-shard2-config52, leaderPayloadVersion=12] 09:29:27,72 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-shard2-config52 from Candidate to Leader 09:29:27,72 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 09:29:27,73 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config52 (Candidate): Received ElectionTimeout 09:29:27,73 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Expecting 2 shard snapshot replies 09:29:27,73 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config52 (Leader): Election: Leader has following peers: [] 09:29:27,73 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-shard1-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 09:29:27,73 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: true, hasLeader : true 09:29:27,73 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: onGetSnapshot 09:29:27,73 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config52: became leader with frontend state for [] 09:29:27,73 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 09:29:27,73 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config52: onGetSnapshot 09:29:27,73 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 09:29:27,73 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 09:29:27,73 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard2-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@2a18991d, electionTerm=1, electionVotedFor=member-1-shard-shard2-config52, ServerConfigPayload=null]] 09:29:27,73 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard1-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 09:29:27,73 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from Candidate to Leader 09:29:27,73 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , registered listener akka://test/user/$$m 09:29:27,73 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard1-config52, leaderId=member-1-shard-shard1-config52, leaderPayloadVersion=12] 09:29:27,74 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-shard1-config52 from Candidate to Leader 09:29:27,74 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config52: All Shards are ready - data store config52 is ready 09:29:27,74 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard1-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@74adfc80, electionTerm=1, electionVotedFor=member-1-shard-shard1-config52, ServerConfigPayload=null]] 09:29:27,74 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: All shard snapshots received 09:29:27,85 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config52: onAddShardReplica: AddShardReplica[ShardName=astronauts] 09:29:27,86 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config52: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-612#147075652 for shard member-1-shard-astronauts-config52 09:29:27,87 AM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-astronauts-config52, persistent : true 09:29:27,88 AM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-astronauts-config52 with journal batch size 1 09:29:27,88 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: Received AddServerReply{status=OK, leaderHint=} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-612#147075652 09:29:27,88 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: Leader shard successfully added the replica shard astronauts 09:29:27,88 AM [test-akka.actor.default-dispatcher-23] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-astronauts-config52/member-1-shard-astronauts-config52-notifier#-915407489 created and ready for shard:member-1-shard-astronauts-config52 09:29:27,88 AM [test-akka.actor.default-dispatcher-24] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-astronauts-config52/shard-member-1:snapshot-read#490889177] 09:29:27,88 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-astronauts-config52 09:29:27,88 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: persisting the shard list [shard2, astronauts, shard1] 09:29:27,89 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 09:29:27,89 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Successfully deleted prior snapshots 09:29:27,89 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : astronauts 09:29:27,89 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-astronauts-config52, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 09:29:27,90 AM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-astronauts-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 09:29:27,90 AM [test-shard-dispatcher-18] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: RaftPolicy used with prev.config org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy, RaftPolicy used with newConfig null 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: Re-initializing to Follower with previous leaderId null 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-astronauts-config52 (Follower): Closing 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52 (Follower): Received TimeoutNow - switching to Candidate 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52 (Candidate): Election: Candidate has following voting peers: [] 09:29:27,90 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: Set currentTerm=1, votedFor=member-1-shard-astronauts-config52 09:29:27,91 AM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-astronauts-config52 (Candidate): Starting new election term 1 09:29:27,91 AM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-astronauts-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 09:29:27,91 AM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-astronauts-config52 (Follower): Closing 09:29:27,91 AM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 09:29:27,91 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 09:29:27,91 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [astronauts] 09:29:27,91 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config52 is initialized 09:29:27,91 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config52 is initialized - notifying 1 OnShardInitialized callbacks 09:29:27,91 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from null to Follower 09:29:27,91 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from Follower to Candidate 09:29:27,91 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 09:29:27,91 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52 (Candidate): Received ElectionTimeout 09:29:27,91 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52 (Leader): Election: Leader has following peers: [] 09:29:27,91 AM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-astronauts-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 09:29:27,91 AM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: true, hasLeader : true 09:29:27,92 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: became leader with frontend state for [] 09:29:27,92 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Expecting 3 shard snapshot replies 09:29:27,92 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from Candidate to Leader 09:29:27,92 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: onGetSnapshot 09:29:27,92 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: onGetSnapshot 09:29:27,92 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 09:29:27,92 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , registered listener akka://test/user/$$m 09:29:27,92 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 09:29:27,92 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-astronauts-config52, leaderId=member-1-shard-astronauts-config52, leaderPayloadVersion=12] 09:29:27,92 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-astronauts-config52 from Candidate to Leader 09:29:27,92 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 09:29:27,92 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config52: All Shards are ready - data store config52 is ready 09:29:27,92 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 09:29:27,92 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: onGetSnapshot 09:29:27,92 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 09:29:27,92 AM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard1-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 09:29:27,92 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard2-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@28d356eb, electionTerm=1, electionVotedFor=member-1-shard-shard2-config52, ServerConfigPayload=null]] 09:29:27,92 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-astronauts-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@65f129d1, electionTerm=1, electionVotedFor=member-1-shard-astronauts-config52, ServerConfigPayload=null]] 09:29:27,92 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard1-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@792f571e, electionTerm=1, electionVotedFor=member-1-shard-shard1-config52, ServerConfigPayload=null]] 09:29:27,92 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: All shard snapshots received 09:29:27,93 AM [main] [INFO] ShardManagerTest - testGetSnapshot ending 09:29:27,196 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$m] 09:29:27,207 AM [test-shard-dispatcher-20] [INFO] Shard - Stopping Shard member-1-shard-astronauts-config52 09:29:27,207 AM [test-shard-dispatcher-20] [DEBUG] MessageSlicer - member-1-shard-astronauts-config52 (Leader)_slicer-id-224: Closing 09:29:27,207 AM [test-shard-dispatcher-16] [INFO] Shard - Stopping Shard member-1-shard-shard2-config52 09:29:27,207 AM [test-shard-dispatcher-16] [DEBUG] MessageSlicer - member-1-shard-shard2-config52 (Leader)_slicer-id-221: Closing 09:29:27,208 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-astronauts-config52: Aborting 0 pending queued transactions 09:29:27,208 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-shard2-config52: Aborting 0 pending queued transactions 09:29:27,208 AM [test-shard-dispatcher-21] [INFO] Shard - Stopping Shard member-1-shard-shard1-config52 09:29:27,208 AM [test-shard-dispatcher-21] [DEBUG] MessageSlicer - member-1-shard-shard1-config52 (Leader)_slicer-id-222: Closing 09:29:27,208 AM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config52: Aborting 0 pending queued transactions 09:29:27,210 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - Stopping ShardManager shard-manager-config52 09:29:27,211 AM [main] [INFO] ShardManagerTest - testWhenMultipleShardsPresentSyncStatusMustBeTrueForAllShards starting 09:29:27,214 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config53 09:29:27,216 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - Recovery complete : shard-manager-config53 09:29:27,216 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManager - shard-manager-config53: Creating local shard: member-1-shard-default-config53 09:29:27,216 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManager - shard-manager-config53: Creating local shard: member-1-shard-astronauts-config53 09:29:27,315 AM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-default-config53 from Follower to Leader 09:29:27,315 AM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-astronauts-config53 from Follower to Leader 09:29:27,315 AM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-astronauts-config53 from Leader to Follower 09:29:27,315 AM [main] [INFO] ShardManager - shard-manager-config53 Received follower initial sync status for member-1-shard-astronauts-config53 status sync done true 09:29:27,315 AM [main] [INFO] ShardManagerTest - testWhenMultipleShardsPresentSyncStatusMustBeTrueForAllShards ending 09:29:27,417 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$n] 09:29:27,427 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config53 09:29:27,429 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithFindPrimaryTimeout starting 09:29:27,431 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config54 09:29:27,434 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config54 09:29:27,442 AM [test-akka.actor.default-dispatcher-24] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:27,442 AM [test-akka.actor.default-dispatcher-24] [INFO] ShardManager - shard-manager-config54: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://non-existent@127.0.0.1:5 09:29:27,442 AM [test-akka.actor.default-dispatcher-24] [INFO] ShardManager - shard-manager-config54: All Shards are ready - data store config54 is ready 09:29:27,442 AM [test-akka.actor.default-dispatcher-24] [DEBUG] ShardManager - shard-manager-config54: onAddShardReplica: AddShardReplica[ShardName=astronauts] 09:29:27,443 AM [test-akka.actor.default-dispatcher-24] [DEBUG] ShardManager - shard-manager-config54: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=true] 09:29:27,443 AM [test-akka.actor.default-dispatcher-24] [DEBUG] ShardManager - shard-manager-config54: findPrimary for astronauts forwarding to remote ShardManager akka://non-existent@127.0.0.1:5/user/shardmanager-config54, visitedAddresses: [akka://test@127.0.0.1:2550/user/shardmanager-config54] 09:29:27,460 AM [test-akka.actor.default-dispatcher-24] [WARN] Materializer - [outbound connection to [akka://non-existent@127.0.0.1:5], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1/:5,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused 09:29:27,462 AM [test-akka.actor.default-dispatcher-24] [WARN] Materializer - [outbound connection to [akka://non-existent@127.0.0.1:5], control stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1/:5,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused 09:29:27,654 AM [test-akka.actor.default-dispatcher-24] [DEBUG] ShardManager - shard-manager-config54: Received failure from FindPrimary for shard astronauts akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://test/user/shardmanager-config54#60014211]] after [200 ms]. Message of type [org.opendaylight.controller.cluster.datastore.messages.FindPrimary]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. 09:29:27,654 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithFindPrimaryTimeout ending 09:29:27,756 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/shardmanager-config54#60014211] 09:29:27,767 AM [test-akka.actor.default-dispatcher-24] [INFO] ShardManager - Stopping ShardManager shard-manager-config54 09:29:27,769 AM [main] [INFO] ShardManagerTest - testOnCreateShard starting 09:29:27,771 AM [test-akka.actor.default-dispatcher-24] [INFO] ShardManager - Starting ShardManager shard-manager-config55 09:29:27,772 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config55 09:29:27,772 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 09:29:27,782 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config55: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@42ddcbbf] 09:29:27,782 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config55 doCreateShard: shardId: member-1-shard-foo-config55, memberNames: [MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], peerAddresses: {member-6-shard-foo-config55=null, member-5-shard-foo-config55=null}, isActiveMember: true 09:29:27,783 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config55: findLocalShard : foo 09:29:27,783 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config55: Scheduling 300000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config55, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 09:29:27,786 AM [test-shard-dispatcher-20] [INFO] Shard - Shard created : member-1-shard-foo-config55, persistent : false 09:29:27,786 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$C/member-1-shard-foo-config55/shard-member-1:snapshot-read#280892548] 09:29:27,786 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$C/member-1-shard-foo-config55/member-1-shard-foo-config55-notifier#541875398 created and ready for shard:member-1-shard-foo-config55 09:29:27,788 AM [test-shard-dispatcher-20] [INFO] Shard - Starting recovery for member-1-shard-foo-config55 with journal batch size 1 09:29:27,795 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config55: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 09:29:27,795 AM [test-shard-dispatcher-20] [INFO] Shard - member-1-shard-foo-config55: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:29:27,795 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManager - shard-manager-config55: Initializing shard [foo] 09:29:27,795 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardInformation - Shard member-1-shard-foo-config55 is initialized 09:29:27,795 AM [test-shard-dispatcher-20] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config55: onLeadershipChange, isLeader: false, hasLeader : false 09:29:27,795 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardInformation - Shard member-1-shard-foo-config55 is initialized - notifying 1 OnShardInitialized callbacks 09:29:27,795 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config55: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config55 is no longer the leader 09:29:27,795 AM [test-akka.actor.default-dispatcher-19] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config55 , received role change from null to Follower 09:29:27,795 AM [test-akka.actor.default-dispatcher-19] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config55 , registered listener akka://test/user/$C 09:29:27,795 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - shard-manager-config55: Received role changed for member-1-shard-foo-config55 from null to Follower 09:29:27,795 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManager - shard-manager-config55: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@42ddcbbf] 09:29:27,796 AM [test-akka.actor.default-dispatcher-19] [DEBUG] ShardManager - shard-manager-config55: Shard foo already exists 09:29:27,796 AM [main] [INFO] ShardManagerTest - testOnCreateShard ending 09:29:27,897 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$C#914792957] 09:29:27,908 AM [test-shard-dispatcher-20] [INFO] Shard - Stopping Shard member-1-shard-foo-config55 09:29:27,908 AM [test-shard-dispatcher-20] [DEBUG] MessageAssembler - member-1-shard-foo-config55 (Follower): Closing 09:29:27,908 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-foo-config55: Aborting 0 pending queued transactions 09:29:27,908 AM [test-akka.actor.default-dispatcher-24] [INFO] ShardManager - Stopping ShardManager shard-manager-config55 09:29:27,910 AM [test-akka.actor.default-dispatcher-24] [INFO] ShardManager - Stopping ShardManager shard-manager-config1 09:29:27,910 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - Stopping ShardManager shard-manager-config35 09:29:27,910 AM [test-akka.actor.default-dispatcher-24] [INFO] ShardManager - Stopping ShardManager shard-manager-config43 09:29:27,910 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config11 09:29:27,910 AM [test-akka.actor.default-dispatcher-24] [INFO] ShardManager - Stopping ShardManager shard-manager-config46 09:29:27,910 AM [test-akka.actor.default-dispatcher-19] [INFO] ShardManager - Stopping ShardManager shard-manager-config21 09:29:27,911 AM [test-akka.actor.default-dispatcher-17] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:29:27,911 AM [test-akka.actor.default-dispatcher-17] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [WARN] [04/25/2024 09:29:27.922] [test-akka.actor.internal-dispatcher-3] [Cluster(akka://test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config