10:44:09,282 AM [test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 10:44:09,312 AM [test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://test@127.0.0.1:2565] with UID [5454325229902586913] 10:44:09,312 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Starting up, Akka version [2.6.21] ... 10:44:09,317 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Registered cluster JMX MBean [akka:type=Cluster] 10:44:09,317 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Started up successfully 10:44:09,331 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:09,331 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:09,539 AM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$a#1949447638] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:09,545 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config1 10:44:09,611 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config1 10:44:09,611 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Creating local shard: member-1-shard-default-config1 10:44:09,627 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:09,627 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config1 10:44:09,627 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Initializing shard [default] 10:44:09,627 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config1 is initialized 10:44:09,628 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config1: Received role changed for member-1-shard-default-config1 from null to Follower 10:44:09,628 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: onChangeShardServersVotingStatus: ChangeShardMembersVotingStatus [shardName=default, meberVotingStatusMap={member-2=true}] 10:44:09,630 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: findLocalShard : default 10:44:09,637 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Sending ChangeServersVotingStatus message ChangeServersVotingStatus [serverVotingStatusMap={member-2-shard-default-config1=true}, serversVisited=[]] to local shard akka://test/user/member-1-shard-default-config1 10:44:09,638 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config1: Received ServerChangeReply{status=NO_LEADER} from local shard akka://test/user/member-1-shard-default-config1 10:44:09,638 AM [test-akka.actor.default-dispatcher-5] [WARN] ShardManager - shard-manager-config1: ChangeServersVotingStatus failed for shard default with status NO_LEADER 10:44:09,793 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config1#1729268964] 10:44:09,809 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForShardLeader starting 10:44:09,813 AM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$b#-985991753] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:09,814 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config2 10:44:09,816 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config2 10:44:09,816 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config2: Creating local shard: member-1-shard-default-config2 10:44:09,824 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:09,825 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config2 10:44:09,828 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config2: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:09,834 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config2: Scheduling 10000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config2, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 10:44:09,975 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config2: Initializing shard [default] 10:44:09,975 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized 10:44:09,976 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized - notifying 1 OnShardInitialized callbacks 10:44:10,126 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config2: Received role changed for member-1-shard-default-config2 from Candidate to Leader 10:44:10,126 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized - notifying 1 OnShardInitialized callbacks 10:44:10,279 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config2: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config2, leaderId=member-1-shard-default-config2, leaderPayloadVersion=13] 10:44:10,279 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is ready - notifying 1 OnShardInitialized callbacks 10:44:10,279 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config2: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#1416989559, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$LtBPDnsk@ce46635] 10:44:10,280 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config2: All Shards are ready - data store config2 is ready 10:44:10,480 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForShardLeader ending 10:44:10,582 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$b#-985991753] 10:44:10,593 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config2 10:44:10,607 AM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/$$a] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:10,608 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config3 10:44:10,610 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config3 10:44:10,610 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config3: Creating local shard: member-1-shard-default-config3 10:44:10,610 AM [main] [INFO] ShardManager - shard-manager-config3: Received role changed for member-1-shard-default-config3 from Follower to Candidate 10:44:10,610 AM [main] [INFO] ShardManager - shard-manager-config3 Received follower initial sync status for member-1-shard-default-config3 status sync done true 10:44:10,712 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$a] 10:44:10,724 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config3 10:44:10,739 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 10:44:10,758 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [4189508021772747108] 10:44:10,758 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 10:44:10,768 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:10,768 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 10:44:10,781 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:10,781 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:10,826 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config4 10:44:10,827 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config4] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:10,827 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 10:44:10,827 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 10:44:10,827 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 10:44:10,853 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - Recovery complete : shard-manager-config4 10:44:10,853 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-1-shard-default-config4 10:44:10,854 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-1-shard-astronauts-config4 10:44:10,855 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 10:44:10,855 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 10:44:10,855 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 10:44:10,865 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 10:44:10,888 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-8348978096016862386] 10:44:10,888 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 10:44:10,888 AM [cluster-test-akka.actor.default-dispatcher-6] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:10,888 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 10:44:10,902 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:10,902 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:10,909 AM [main] [ERROR] ShardManagerTest - Mock Shard Leader Actor : TestActor[akka://cluster-test/user/member-2-shard-default-config4] 10:44:10,924 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config4 10:44:10,924 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config4] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:10,933 AM [main] [ERROR] ShardManagerTest - Forwarding actor : Actor[akka://cluster-test/user/shardmanager-config4/member-2-shard-default-config4#-147577810] 10:44:10,942 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config4 10:44:10,942 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-2-shard-default-config4 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config4 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config4: Initializing shard [default] 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-2-shard-default-config4 is initialized 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config4 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config4: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config4, leaderId=member-2-shard-default-config4, leaderPayloadVersion=12] 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 to akka://cluster-test/user/member-1-shard-default-config 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config4: Received role changed for member-2-shard-default-config4 from Candidate to Leader 10:44:10,946 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 10:44:10,947 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: updated peer member-1-shard-default-config4 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 on shard member-1-shard-default-config4 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#-1282337773] 10:44:10,947 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-1-shard-astronauts-config4 10:44:10,947 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 10:44:10,947 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 to akka://cluster-test/user/member-1-shard-default-config 10:44:10,947 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: updated peer member-1-shard-astronauts-config4 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 on shard member-1-shard-astronauts-config4 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#-1282337773] 10:44:10,947 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: Initializing shard [default] 10:44:10,947 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-1-shard-default-config4 is initialized 10:44:10,947 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config4: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config4, leaderId=member-2-shard-default-config4, leaderPayloadVersion=12] 10:44:10,947 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config4: Received role changed for member-1-shard-default-config4 from Candidate to Follower 10:44:11,402 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 10:44:11,456 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 10:44:11,463 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 10:44:11,463 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 10:44:11,463 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 to akka://cluster-test/user/member-2-shard-default-config4 10:44:11,463 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 10:44:11,792 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 10:44:11,792 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:11,793 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 10:44:11,793 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 to akka://cluster-test/user/member-1-shard-default-config 10:44:11,793 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-astronauts-config4 10:44:11,793 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-astronauts-config4 to akka://cluster-test/user/member-1-shard-default-config 10:44:11,794 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: onRemoveShardReplica: RemoveShardReplica [shardName=default, memberName=MemberName{name=member-1}] 10:44:11,797 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:11,799 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4, primaryVersion=12] 10:44:11,801 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config4: Sending RemoveServer message to peer akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 for shard member-1-shard-default-config4 10:44:11,849 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config4: Received RemoveServerReply{status=OK, leaderHint=member-2-shard-default-config4} for shard default 10:44:11,849 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config4: Leader shard successfully removed the replica shard default 10:44:11,937 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:11,937 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 10:44:11,937 AM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 to akka://cluster-test/user/member-2-shard-default-config4 10:44:11,937 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 10:44:11,952 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shard-manager-config4 10:44:11,958 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:11,958 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:11,997 AM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:11,997 AM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:12,6 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 10:44:12,33 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config4 10:44:12,38 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:12,38 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:13,59 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 10:44:13.064] [cluster-test-akka.actor.internal-dispatcher-4] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:13,85 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config5 10:44:13,85 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$c#1202617576] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:13,87 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config5 10:44:13,87 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config5: Creating local shard: member-1-shard-default-config5 10:44:13,95 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:13,95 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config5 10:44:13,96 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config5: Initializing shard [default] 10:44:13,96 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config5 is initialized 10:44:13,250 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$c#1202617576] 10:44:13,261 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config5 10:44:13,267 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/$$b] to Actor[akka://test/user/termination-monitor] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:13,267 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config6 10:44:13,269 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config6 10:44:13,270 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config6: Creating local shard: member-1-shard-default-config6 10:44:13,270 AM [main] [INFO] ShardManager - shard-manager-config6: Received role changed for member-1-shard-default-config6 from null to Follower 10:44:13,281 AM [main] [INFO] ShardManager - shard-manager-config6: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 10:44:13,281 AM [main] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config6 with address akka://test/user/shardmanager-config6/member-2-shard-default-config6 10:44:13,281 AM [main] [INFO] ShardManager - shard-manager-config6: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config6, leaderId=member-2-shard-default-config6, leaderPayloadVersion=13] 10:44:13,281 AM [main] [INFO] ShardManager - shard-manager-config6: All Shards are ready - data store config6 is ready 10:44:13,383 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$b] 10:44:13,394 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config6 10:44:13,401 AM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$d#704049401] to Actor[akka://test/user/termination-monitor] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:13,401 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config7 10:44:13,403 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config7 10:44:13,412 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config7: onRemoveShardReplica: RemoveShardReplica [shardName=model-inventory, memberName=MemberName{name=member-1}] 10:44:13,412 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config7: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=model-inventory, waitUntilReady=true] 10:44:13,413 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config7: No shard found for model-inventory 10:44:13,414 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config7: Failed to find leader for shard model-inventory: received response: {} org.opendaylight.controller.cluster.datastore.exceptions.PrimaryNotFoundException: No primary shard found for model-inventory. at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.findPrimary(ShardManager.java:1051) at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.handleCommand(ShardManager.java:207) at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManagerTest$TestShardManager.handleCommand(ShardManagerTest.java:2199) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) at scala.PartialFunction.applyOrElse(PartialFunction.scala:214) at scala.PartialFunction.applyOrElse$(PartialFunction.scala:213) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:537) at akka.actor.Actor.aroundReceive$(Actor.scala:535) at akka.persistence.AbstractPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:295) at akka.persistence.Eventsourced$$anon$4.stateReceive(Eventsourced.scala:917) at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:245) at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:244) at akka.persistence.AbstractPersistentActor.aroundReceive(PersistentActor.scala:295) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579) at akka.actor.ActorCell.invoke(ActorCell.scala:547) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) at akka.dispatch.Mailbox.run(Mailbox.scala:231) at akka.dispatch.Mailbox.exec(Mailbox.scala:243) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) 10:44:13,516 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$d#704049401] 10:44:13,527 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config7 10:44:13,530 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingLocalReplicaLeader starting 10:44:13,533 AM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$e#1235608360] to Actor[akka://test/user/termination-monitor] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:13,533 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config8 10:44:13,534 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config8 10:44:13,534 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config8: Creating local shard: member-1-shard-default-config8 10:44:13,544 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:13,544 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config8 10:44:13,544 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config8: Initializing shard [default] 10:44:13,544 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config8 is initialized 10:44:13,544 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config8: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config8, leaderId=member-1-shard-default-config8, leaderPayloadVersion=13] 10:44:13,544 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config8: Received role changed for member-1-shard-default-config8 from Candidate to Leader 10:44:13,544 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config8: All Shards are ready - data store config8 is ready 10:44:13,545 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config8: onAddShardReplica: AddShardReplica[ShardName=default] 10:44:13,546 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config8: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:13,546 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config8: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#-2037080226, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$LtBPDnsk@5070d54c] 10:44:13,547 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config8: Local shard default already exists 10:44:13,547 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config8: findLocalShard : default 10:44:13,547 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingLocalReplicaLeader ending 10:44:13,650 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$e#1235608360] 10:44:13,661 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config8 10:44:13,663 AM [main] [INFO] ShardManagerTest - testOnReceiveFindLocalShardWaitForShardInitialized starting 10:44:13,667 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$f#-1189017884] to Actor[akka://test/user/termination-monitor] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:13,667 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config9 10:44:13,669 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config9 10:44:13,670 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config9: Creating local shard: member-1-shard-default-config9 10:44:13,682 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:13,682 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config9 10:44:13,682 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config9: findLocalShard : default 10:44:13,682 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config9: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config9, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 10:44:13,683 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config9: Initializing shard [default] 10:44:13,683 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config9 is initialized 10:44:13,683 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config9 is initialized - notifying 1 OnShardInitialized callbacks 10:44:13,689 AM [main] [INFO] ShardManagerTest - testOnReceiveFindLocalShardWaitForShardInitialized starting 10:44:13,791 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$f#-1189017884] 10:44:13,803 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config9 10:44:13,806 AM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberUnreachableAndLeadershipChange starting 10:44:13,832 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 10:44:13,850 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [6774097488356024854] 10:44:13,850 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 10:44:13,865 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 10:44:13,865 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 10:44:13,865 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:13,865 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:13,880 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config10 10:44:13,886 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 10:44:13,886 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 10:44:13,886 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 10:44:13,887 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config10] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:13,890 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config10 10:44:13,890 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config10: Creating local shard: member-1-shard-default-config10 10:44:13,891 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 10:44:13,891 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 10:44:13,894 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 10:44:13,928 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-1202529756272686625] 10:44:13,928 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 10:44:13,928 AM [cluster-test-akka.actor.default-dispatcher-6] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:13,928 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 10:44:13,953 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:13,953 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:13,960 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config10] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:13,966 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Starting ShardManager shard-manager-config10 10:44:13,970 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:13,970 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config10 10:44:13,970 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 10:44:13,970 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 to akka://cluster-test/user/member-1-shard-default-config 10:44:13,971 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config10: updated peer member-1-shard-default-config10 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 on shard member-1-shard-default-config10 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#-1348888753] 10:44:13,971 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config10: Initializing shard [default] 10:44:13,971 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config10 is initialized 10:44:13,971 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config10, leaderId=member-2-shard-default-config10, leaderPayloadVersion=13] 10:44:13,971 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config10: Received role changed for member-1-shard-default-config10 from Candidate to Follower 10:44:13,974 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - Recovery complete : shard-manager-config10 10:44:13,975 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config10: Creating local shard: member-2-shard-default-config10 10:44:13,976 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:13,976 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config10 10:44:13,976 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config10: Initializing shard [default] 10:44:13,976 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardInformation - Shard member-2-shard-default-config10 is initialized 10:44:13,976 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config10, leaderId=member-2-shard-default-config10, leaderPayloadVersion=13] 10:44:13,976 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config10: Received role changed for member-2-shard-default-config10 from Candidate to Leader 10:44:13,976 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 10:44:14,48 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 10:44:14,52 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 10:44:14,53 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 10:44:14,53 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 10:44:14,53 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 to akka://cluster-test/user/member-2-shard-default-config 10:44:14,53 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 10:44:14,868 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:14,869 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 10:44:14,869 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 to akka://cluster-test/user/member-1-shard-default-config 10:44:14,869 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 10:44:14,869 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 10:44:14,869 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:14,869 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config10: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10, primaryVersion=13] 10:44:14,871 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 10:44:14,871 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config10 as unavailable. 10:44:14,871 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:14,872 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config10: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config10, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config10, activeMember=true] 10:44:14,970 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:14,970 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 10:44:14,970 AM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 to akka://cluster-test/user/member-2-shard-default-config 10:44:14,970 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 10:44:16,85 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config10: Received ShardNotInitializedTimeout message for shard default 10:44:16,85 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config10: Returning NoShardLeaderException for shard default 10:44:16,86 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config10, leaderId=member-1-shard-default-config10, leaderPayloadVersion=13] 10:44:16,87 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 10:44:16,87 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config10: Received role changed for member-1-shard-default-config10 from Follower to Leader 10:44:16,87 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 10:44:16,87 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:16,87 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config10: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/member-1-shard-default-config#-1348888753, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$LtBPDnsk@61d70a1e] 10:44:16,87 AM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberUnreachableAndLeadershipChange ending 10:44:16,190 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - Stopping ShardManager shard-manager-config10 10:44:16,192 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:16,192 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:16,207 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:16,207 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:16,209 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 10:44:16,224 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - Stopping ShardManager shard-manager-config10 10:44:16,225 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:16,225 AM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:17,248 AM [cluster-test-akka.actor.default-dispatcher-16] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 10:44:17.250] [cluster-test-akka.actor.internal-dispatcher-3] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:17,265 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config11 10:44:17,265 AM [test-akka.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/shardmanager-config11] to Actor[akka://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:17,268 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config11 10:44:17,288 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:17,288 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config11: onAddShardReplica: AddShardReplica[ShardName=astronauts] 10:44:17,291 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config11: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-530#1890398186 for shard member-1-shard-astronauts-config11 10:44:17,303 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config11: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-2}] 10:44:17,304 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config11: A shard replica operation for astronauts is already in progress 10:44:17,409 AM [main] [INFO] ShardManagerTest - testPerShardDatastoreContext starting 10:44:17,418 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config12 10:44:17,421 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config12 10:44:17,421 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config12: Creating local shard: member-1-shard-default-config12 10:44:17,422 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config12: Creating local shard: member-1-shard-topology-config12 10:44:17,423 AM [test-akka.actor.default-dispatcher-15] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$g#-1743727480] to Actor[akka://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:17,429 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:17,429 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config12 10:44:17,429 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-topology-config12 10:44:17,431 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-default-config12 10:44:17,431 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-topology-config12 10:44:17,482 AM [main] [INFO] ShardManagerTest - testPerShardDatastoreContext ending 10:44:17,584 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/default1#-1050015446] 10:44:17,596 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/topology2#-389641120] 10:44:17,596 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$g#-1743727480] 10:44:17,596 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config12 10:44:17,599 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForLocalLeaderShard starting 10:44:17,602 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config13 10:44:17,604 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config13 10:44:17,604 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config13: Creating local shard: member-1-shard-default-config13 10:44:17,613 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:17,613 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config13 10:44:17,613 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config13: Initializing shard [default] 10:44:17,613 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-default-config13 is initialized 10:44:17,613 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config13: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config13, leaderId=member-1-shard-default-config13, leaderPayloadVersion=13] 10:44:17,664 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config13: Received role changed for member-1-shard-default-config13 from Candidate to Leader 10:44:17,664 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config13: All Shards are ready - data store config13 is ready 10:44:17,665 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config13: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 10:44:17,665 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config13: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#1453833456, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$LtBPDnsk@31d3b794] 10:44:17,665 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForLocalLeaderShard ending 10:44:17,767 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$h#-453216552] 10:44:17,779 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config13 10:44:17,782 AM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberWithNameContainedInLeaderIdUnreachable starting 10:44:17,792 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 10:44:17,809 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2562] with UID [-2519385933872390509] 10:44:17,809 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Starting up, Akka version [2.6.21] ... 10:44:17,812 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Registered cluster JMX MBean [akka:type=Cluster] 10:44:17,812 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Started up successfully 10:44:17,822 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:17,822 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:17,828 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Node [akka://cluster-test@127.0.0.1:2562] is JOINING itself (with roles [member-256, dc-default], version [0.0.0]) and forming new cluster 10:44:17,828 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - is the new leader among reachable nodes (more leaders may exist) 10:44:17,828 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Leader is moving node [akka://cluster-test@127.0.0.1:2562] to [Up] 10:44:17,833 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config14 10:44:17,833 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config14] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:17,852 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 10:44:17,855 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - Recovery complete : shard-manager-config14 10:44:17,855 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config14: Creating local shard: member-256-shard-default-config14 10:44:17,856 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-256}, address: akka://cluster-test@127.0.0.1:2562 10:44:17,856 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 10:44:17,870 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-5863789612663307087] 10:44:17,871 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 10:44:17,876 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:17,876 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 10:44:17,876 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:17,876 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:17,890 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config14] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:17,892 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config14 10:44:17,899 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:17,899 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - Creating Shard member-256-shard-default-config14 10:44:17,899 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 10:44:17,899 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 to akka://cluster-test/user/member-256-shard-default-config 10:44:17,899 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config14: updated peer member-256-shard-default-config14 on member MemberName{name=member-256} with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 on shard member-256-shard-default-config14 whose actor address is Actor[akka://cluster-test/user/member-256-shard-default-config#1489030497] 10:44:17,899 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config14: Initializing shard [default] 10:44:17,899 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardInformation - Shard member-256-shard-default-config14 is initialized 10:44:17,899 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config14: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-256-shard-default-config14, leaderId=member-256-shard-default-config14, leaderPayloadVersion=13] 10:44:17,900 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config14: Received role changed for member-256-shard-default-config14 from Candidate to Leader 10:44:17,900 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 10:44:17,905 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config14 10:44:17,905 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config14: Creating local shard: member-2-shard-default-config14 10:44:17,906 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:17,906 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config14 10:44:17,906 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config14: Initializing shard [default] 10:44:17,906 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-2-shard-default-config14 is initialized 10:44:17,906 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config14: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config14, leaderId=member-256-shard-default-config14, leaderPayloadVersion=13] 10:44:17,906 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config14: Received role changed for member-2-shard-default-config14 from Candidate to Follower 10:44:17,956 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 10:44:17,958 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2562] 10:44:17,961 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-256}, address: akka://cluster-test@127.0.0.1:2562 10:44:17,961 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 10:44:17,961 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 to akka://cluster-test/user/member-2-shard-default-config 10:44:17,961 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 10:44:18,840 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 10:44:18,840 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:18,840 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 10:44:18,840 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 to akka://cluster-test/user/member-256-shard-default-config 10:44:18,840 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 10:44:18,841 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config14: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:18,841 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config14: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2562/user/member-256-shard-default-config#1489030497, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$LtBPDnsk@7ead3345] 10:44:18,842 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 10:44:18,842 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config14: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:18,842 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config14: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2562/user/member-256-shard-default-config#1489030497, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$LtBPDnsk@7ead3345] 10:44:18,845 AM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberWithNameContainedInLeaderIdUnreachable ending 10:44:18,900 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:18,900 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 10:44:18,900 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 to akka://cluster-test/user/member-2-shard-default-config 10:44:18,900 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 10:44:18,948 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - Stopping ShardManager shard-manager-config14 10:44:18,948 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:18,948 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:18,967 AM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:18,967 AM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:18,969 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 10:44:18,984 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config14 10:44:18,985 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:18,985 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:19,911 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - is the new leader among reachable nodes (more leaders may exist) 10:44:20,2 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 10:44:20.005] [cluster-test-akka.actor.internal-dispatcher-3] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:20,26 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config15 10:44:20,30 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config15 10:44:20,30 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config15: Creating local shard: member-1-shard-default-config15 10:44:20,31 AM [main] [INFO] ShardManager - shard-manager-config15: Received role changed for member-1-shard-default-config15 from Follower to Leader 10:44:20,133 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$c] 10:44:20,144 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config15 10:44:20,151 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config16 10:44:20,152 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config16 10:44:20,152 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config16: Creating local shard: member-1-shard-default-config16 10:44:20,153 AM [main] [INFO] ShardManager - shard-manager-config16: Received role changed for member-1-shard-default-config16 from Candidate to Leader 10:44:20,153 AM [main] [INFO] ShardManager - shard-manager-config16: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config16, leaderId=member-1-shard-default-config16, leaderPayloadVersion=13] 10:44:20,153 AM [main] [INFO] ShardManager - shard-manager-config16: All Shards are ready - data store config16 is ready 10:44:20,255 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$d] 10:44:20,266 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config16 10:44:20,268 AM [main] [INFO] ShardManagerTest - testOnCreateShardWithLocalMemberNotInShardConfig starting 10:44:20,271 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config17 10:44:20,273 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config17 10:44:20,273 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:20,288 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config17: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@62187ba4] 10:44:20,290 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config17 doCreateShard: shardId: member-1-shard-foo-config17, memberNames: [MemberName{name=member-5}, MemberName{name=member-6}], peerAddresses: {}, isActiveMember: false 10:44:20,300 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config17: findLocalShard : foo 10:44:20,300 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config17: Scheduling 60000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config17, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=false] 10:44:20,310 AM [test-shard-dispatcher-16] [INFO] Shard - Shard created : member-1-shard-foo-config17, persistent : true 10:44:20,328 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$i/member-1-shard-foo-config17/member-1-shard-foo-config17-notifier#-421477565 created and ready for shard:member-1-shard-foo-config17 10:44:20,337 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$i/member-1-shard-foo-config17/shard-member-1:snapshot-read#1900344690] 10:44:20,343 AM [test-shard-dispatcher-16] [INFO] Shard - Starting recovery for member-1-shard-foo-config17 with journal batch size 1 10:44:20,365 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-foo-config17: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 10:44:20,369 AM [test-shard-dispatcher-16] [INFO] Shard - member-1-shard-foo-config17: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 10:44:20,370 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config17: Initializing shard [foo] 10:44:20,370 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-foo-config17 is initialized 10:44:20,370 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-foo-config17 is initialized - notifying 1 OnShardInitialized callbacks 10:44:20,370 AM [main] [INFO] ShardManagerTest - testOnCreateShardWithLocalMemberNotInShardConfig ending 10:44:20,377 AM [test-shard-dispatcher-16] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config17: onLeadershipChange, isLeader: false, hasLeader : false 10:44:20,378 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-foo-config17: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config17 is no longer the leader 10:44:20,378 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config17 , received role change from null to Follower 10:44:20,382 AM [test-shard-dispatcher-16] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 10:44:20,384 AM [test-akka.actor.default-dispatcher-15] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config17 , registered listener akka://test/user/$i 10:44:20,384 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config17: Received role changed for member-1-shard-foo-config17 from null to Follower 10:44:20,472 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$i#1570766526] 10:44:20,484 AM [test-shard-dispatcher-16] [INFO] Shard - Stopping Shard member-1-shard-foo-config17 10:44:20,484 AM [test-shard-dispatcher-16] [DEBUG] MessageAssembler - member-1-shard-foo-config17 (Follower): Closing 10:44:20,485 AM [test-shard-dispatcher-16] [DEBUG] Shard - member-1-shard-foo-config17: Aborting 0 pending queued transactions 10:44:20,486 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config17 10:44:20,488 AM [main] [INFO] ShardManagerTest - testServerRemovedShardActorNotRunning starting 10:44:20,491 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config18 10:44:20,493 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config18 10:44:20,494 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config18: Creating local shard: member-1-shard-default-config18 10:44:20,494 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config18: Creating local shard: member-1-shard-people-config18 10:44:20,505 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: findLocalShard : people 10:44:20,505 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: Returning NotInitializedException for shard people 10:44:20,506 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: findLocalShard : default 10:44:20,506 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: Returning NotInitializedException for shard default 10:44:20,508 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18 : Local Shard replica for shard default has been removed 10:44:20,508 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: persisting the shard list [people] 10:44:20,516 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 10:44:20,516 AM [main] [INFO] ShardManagerTest - testServerRemovedShardActorNotRunning ending 10:44:20,523 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config18: Successfully deleted prior snapshots 10:44:20,618 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$e] 10:44:20,629 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config18 10:44:20,636 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config19 10:44:20,638 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config19 10:44:20,638 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config19: Creating local shard: member-1-shard-default-config19 10:44:20,648 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:20,648 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config19 10:44:20,648 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config19: Initializing shard [default] 10:44:20,648 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-default-config19 is initialized 10:44:20,648 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config19: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 10:44:20,648 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config19: Returning NoShardLeaderException for shard default 10:44:20,750 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$j#-932466620] 10:44:20,761 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config19 10:44:20,764 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForFollowerShardWithNoInitialLeaderId starting 10:44:20,768 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config20 10:44:20,770 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config20 10:44:20,770 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config20: Creating local shard: member-1-shard-default-config20 10:44:20,779 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:20,779 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config20 10:44:20,779 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: Initializing shard [default] 10:44:20,779 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config20 is initialized 10:44:20,779 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config20: Received role changed for member-1-shard-default-config20 from Candidate to Follower 10:44:20,779 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 10:44:20,779 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: Returning NoShardLeaderException for shard default 10:44:20,780 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config20: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config20, leaderId=member-1-shard-default-config20, leaderPayloadVersion=13] 10:44:20,780 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config20: All Shards are ready - data store config20 is ready 10:44:20,780 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 10:44:20,780 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config20: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#1835662476, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$LtBPDnsk@603b4891] 10:44:20,780 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForFollowerShardWithNoInitialLeaderId starting 10:44:20,882 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$k#995923512] 10:44:20,893 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config20 10:44:20,899 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config21 10:44:20,901 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config21 10:44:20,920 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:20,920 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config21: onAddShardReplica: AddShardReplica[ShardName=astronauts] 10:44:20,921 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config21: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-548#1991429227 for shard member-1-shard-astronauts-config21 10:44:20,933 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config21: onAddShardReplica: AddShardReplica[ShardName=astronauts] 10:44:20,933 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config21: A shard replica operation for astronauts is already in progress 10:44:21,38 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithCandidateShard starting 10:44:21,41 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config22 10:44:21,43 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config22 10:44:21,43 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config22: Creating local shard: member-1-shard-default-config22 10:44:21,52 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:21,52 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config22 10:44:21,52 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config22: Initializing shard [default] 10:44:21,52 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-default-config22 is initialized 10:44:21,52 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config22: Received role changed for member-1-shard-default-config22 from null to Candidate 10:44:21,52 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config22: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:21,52 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config22: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config22, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Candidate, leaderId=null, activeMember=true] 10:44:22,271 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config22: Received ShardNotInitializedTimeout message for shard default 10:44:22,271 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config22: Returning NoShardLeaderException for shard default 10:44:22,272 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithCandidateShard ending 10:44:22,375 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$l#-1457920525] 10:44:22,386 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config22 10:44:22,389 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithAddServerReplyFailure starting 10:44:22,393 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config23 10:44:22,394 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config23 10:44:22,515 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:22,525 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config23: onAddShardReplica: AddShardReplica[ShardName=astronauts] 10:44:22,526 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config23: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-553#-1586175072 for shard member-1-shard-astronauts-config23 10:44:22,528 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config23: Received AddServerReply{status=TIMEOUT} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-553#-1586175072 10:44:22,528 AM [test-akka.actor.default-dispatcher-15] [WARN] ShardManager - shard-manager-config23: Leader failed to add shard replica astronauts with status TIMEOUT 10:44:22,529 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config23: findLocalShard : astronauts 10:44:22,529 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config23: Local shard astronauts not found - shards present: [] 10:44:22,534 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config23: onAddShardReplica: AddShardReplica[ShardName=astronauts] 10:44:22,535 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config23: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-553#-1586175072 for shard member-1-shard-astronauts-config23 10:44:22,535 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config23: Received AddServerReply{status=NO_LEADER} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-553#-1586175072 10:44:22,535 AM [test-akka.actor.default-dispatcher-15] [WARN] ShardManager - shard-manager-config23: Leader failed to add shard replica astronauts with status NO_LEADER 10:44:22,535 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithAddServerReplyFailure ending 10:44:22,637 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-astronauts-config#206324091] 10:44:22,649 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/shardmanager-config23] 10:44:22,649 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config23 10:44:22,655 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config24 10:44:22,657 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config24 10:44:22,657 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config24: Creating local shard: member-1-shard-default-config24 10:44:22,658 AM [main] [INFO] ShardManager - shard-manager-config24: Received role changed for member-1-shard-default-config24 from Candidate to Follower 10:44:22,658 AM [main] [INFO] ShardManager - shard-manager-config24 Received follower initial sync status for member-1-shard-default-config24 status sync done true 10:44:22,658 AM [main] [INFO] ShardManager - shard-manager-config24 Received follower initial sync status for member-1-shard-default-config24 status sync done false 10:44:22,760 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$f] 10:44:22,771 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config24 10:44:22,777 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config25 10:44:22,779 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config25 10:44:22,779 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config25: Creating local shard: member-1-shard-default-config25 10:44:22,779 AM [main] [INFO] ShardManager - shard-manager-config25: Received role changed for unknown from Candidate to Leader 10:44:22,881 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$g] 10:44:22,893 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config25 10:44:22,896 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithUninitializedShard starting 10:44:22,900 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config26 10:44:22,903 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config26 10:44:22,903 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config26: Creating local shard: member-1-shard-default-config26 10:44:22,911 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:22,911 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config26 10:44:22,911 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config26: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:22,911 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config26: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config26, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 10:44:23,531 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config26: Received ShardNotInitializedTimeout message for shard default 10:44:23,531 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config26: Returning NotInitializedException for shard default 10:44:23,532 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config26: Initializing shard [default] 10:44:23,532 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config26 is initialized 10:44:23,732 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithUninitializedShard ending 10:44:23,834 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$m#1612191865] 10:44:23,846 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config26 10:44:23,852 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config27 10:44:23,854 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config27 10:44:23,854 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config27: Creating local shard: member-1-shard-default-config27 10:44:23,863 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:23,863 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config27 10:44:23,863 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config27: Initializing shard [default] 10:44:23,863 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config27 is initialized 10:44:23,863 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config27: findLocalShard : default 10:44:23,968 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$n#-438926070] 10:44:23,979 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config27 10:44:23,983 AM [main] [INFO] ShardManagerTest - testRegisterForShardLeaderChanges starting 10:44:23,986 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config28 10:44:23,988 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config28 10:44:23,988 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config28: Creating local shard: member-1-shard-default-config28 10:44:23,996 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:23,997 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config28 10:44:23,997 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config28: Initializing shard [default] 10:44:23,997 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config28 is initialized 10:44:23,997 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config28: onRegisterForShardAvailabilityChanges: org.opendaylight.controller.cluster.datastore.shardmanager.RegisterForShardAvailabilityChanges@161ab381 10:44:24,3 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=13] 10:44:24,27 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=13] 10:44:24,527 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=null, leaderPayloadVersion=13] 10:44:24,539 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-2-shard-default-config28, leaderPayloadVersion=13] 10:44:24,550 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=13] 10:44:25,51 AM [main] [INFO] ShardManagerTest - testRegisterForShardLeaderChanges ending 10:44:25,153 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$o#494661624] 10:44:25,164 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config28 10:44:25,170 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config29 10:44:25,176 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config29 10:44:25,176 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config29: Creating local shard: member-1-shard-default-config29 10:44:25,279 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$h] 10:44:25,290 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config29 10:44:25,295 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config30 10:44:25,297 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config30 10:44:25,297 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config30: Creating local shard: member-1-shard-default-config30 10:44:25,306 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:25,306 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config30 10:44:25,306 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config30: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=non-existent, waitUntilReady=false] 10:44:25,306 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config30: No shard found for non-existent 10:44:25,408 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$p#-1123847610] 10:44:25,419 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config30 10:44:25,425 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config31 10:44:25,427 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config31 10:44:25,436 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config31: onAddShardReplica: AddShardReplica[ShardName=model-inventory] 10:44:25,436 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config31: No module configuration exists for shard model-inventory 10:44:25,538 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$q#1974978369] 10:44:25,550 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config31 10:44:25,552 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingReplicaInRemoteShardLeader starting 10:44:25,555 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config32 10:44:25,557 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config32 10:44:25,557 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Creating local shard: member-1-shard-default-config32 10:44:25,566 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:25,566 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config32 10:44:25,566 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Initializing shard [default] 10:44:25,566 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config32 is initialized 10:44:25,566 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config32: Received MemberUp: memberName: MemberName{name=leader-member}, address: akka://test 10:44:25,566 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer leader-member-shard-default-config32 with address akka://test/user/shardmanager-config32/leader-member-shard-default-config32 10:44:25,566 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer leader-member-shard-default-config32 with address akka://test/user/shardmanager-config32/leader-member-shard-default-config32 to akka://test/user/member-1-shard-default-config 10:44:25,566 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config32: Received role changed for member-1-shard-default-config32 from Candidate to Follower 10:44:25,566 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config32: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config32, leaderId=leader-member-shard-default-config32, leaderPayloadVersion=13] 10:44:25,566 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config32: All Shards are ready - data store config32 is ready 10:44:25,567 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 10:44:25,567 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:25,567 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=13] 10:44:25,567 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 10:44:25,568 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Received AddServerReply{status=ALREADY_EXISTS} for shard default from leader akka://test/user/shardmanager-config32/leader-member-shard-default-config32 10:44:25,568 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Local shard default already exists 10:44:25,617 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: findLocalShard : default 10:44:25,617 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 10:44:25,618 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:25,618 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=13] 10:44:25,618 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 10:44:25,619 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config32: Received AddServerReply{status=ALREADY_EXISTS} for shard default from leader akka://test/user/shardmanager-config32/leader-member-shard-default-config32 10:44:25,619 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config32: Local shard default already exists 10:44:25,622 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-default-config32 10:44:25,622 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 10:44:25,622 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:25,622 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=13] 10:44:25,623 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 10:44:25,740 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: AddServer request to akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for default failed akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://test/), Path(/user/shardmanager-config32/leader-member-shard-default-config32)]] after [100 ms]. Message of type [org.opendaylight.controller.cluster.raft.messages.AddServer]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. 10:44:25,742 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config32: findLocalShard : default 10:44:25,742 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingReplicaInRemoteShardLeader ending 10:44:25,844 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/shardmanager-config32] 10:44:25,855 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config32 10:44:25,857 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForRemoteShard starting 10:44:25,865 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 10:44:25,876 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [-5203117368826813447] 10:44:25,876 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 10:44:25,879 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 10:44:25,879 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 10:44:25,879 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:25,879 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:25,897 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 10:44:25,897 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 10:44:25,897 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 10:44:25,897 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config33] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:25,901 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Starting ShardManager shard-manager-config33 10:44:25,903 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config33 10:44:25,903 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 10:44:25,903 AM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-1-shard-default-config33 10:44:25,904 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 10:44:25,904 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 10:44:25,913 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [2516901956282638234] 10:44:25,913 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 10:44:25,916 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:25,916 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 10:44:25,920 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:25,920 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:25,939 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config33] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:25,941 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config33 10:44:25,943 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:25,943 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config33 10:44:25,943 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 10:44:25,943 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 to akka://test/user/member-1-shard-default-config 10:44:25,943 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config33: updated peer member-1-shard-default-config33 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 on shard member-1-shard-default-config33 whose actor address is Actor[akka://test/user/member-1-shard-default-config#-813666911] 10:44:25,947 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Recovery complete : shard-manager-config33 10:44:25,947 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-2-shard-default-config33 10:44:25,948 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-2-shard-astronauts-config33 10:44:25,948 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:25,948 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config33 10:44:25,948 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-2-shard-astronauts-config33 10:44:25,948 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config33: Initializing shard [astronauts] 10:44:25,948 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-2-shard-astronauts-config33 is initialized 10:44:25,948 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config33: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-astronauts-config33, leaderId=member-2-shard-astronauts-config33, leaderPayloadVersion=12] 10:44:25,948 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config33: Received role changed for member-2-shard-astronauts-config33 from Candidate to Leader 10:44:25,996 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 10:44:26,1 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 10:44:26,3 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 10:44:26,3 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 10:44:26,3 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 10:44:26,4 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-astronauts-config33 10:44:26,4 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-astronauts-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 10:44:26,894 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 10:44:26,896 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:26,896 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 10:44:26,896 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 to akka://test/user/member-1-shard-default-config 10:44:26,897 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config33: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=false] 10:44:26,897 AM [cluster-test-akka.actor.default-dispatcher-6] [DEBUG] ShardManager - shard-manager-config33: findPrimary for astronauts forwarding to remote ShardManager akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33, visitedAddresses: [akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33] 10:44:26,904 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config33: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.RemoteFindPrimary [shardName=astronauts, waitUntilReady=false] 10:44:26,904 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config33: Found primary for astronauts: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config#-2033735288, primaryVersion=12] 10:44:26,908 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForRemoteShard ending 10:44:26,936 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:26,936 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 10:44:26,936 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 10:44:26,936 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-astronauts-config33 10:44:26,936 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-astronauts-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 10:44:27,15 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shard-manager-config33 10:44:27,17 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:27,17 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:27,27 AM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:27,27 AM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:27,29 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 10:44:27,45 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config33 10:44:27,46 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:27,46 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:28,69 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 10:44:28.073] [cluster-test-akka.actor.internal-dispatcher-2] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:28,81 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config34 10:44:28,84 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config34 10:44:28,84 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config34: Creating local shard: member-1-shard-default-config34 10:44:28,93 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config34: findLocalShard : default 10:44:28,93 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config34: Returning NotInitializedException for shard default 10:44:28,195 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$r#-2097735992] 10:44:28,206 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config34 10:44:28,212 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config35 10:44:28,214 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config35 10:44:28,214 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config35: Creating local shard: member-1-shard-default-config35 10:44:28,220 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:28,220 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config35 10:44:28,221 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config35: Initializing shard [default] 10:44:28,221 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config35 is initialized 10:44:28,221 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config35: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config35, leaderId=member-1-shard-default-config35, leaderPayloadVersion=13] 10:44:28,221 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config35: Received role changed for member-1-shard-default-config35 from Candidate to Leader 10:44:28,221 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config35: All Shards are ready - data store config35 is ready 10:44:28,221 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config35: onChangeShardServersVotingStatus: ChangeShardMembersVotingStatus [shardName=default, meberVotingStatusMap={member-2=true}] 10:44:28,221 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config35: findLocalShard : default 10:44:28,222 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config35: Sending ChangeServersVotingStatus message ChangeServersVotingStatus [serverVotingStatusMap={member-2-shard-default-config35=true}, serversVisited=[]] to local shard akka://test/user/member-1-shard-default-config35 10:44:28,222 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config35: Received ServerChangeReply{status=OK} from local shard akka://test/user/member-1-shard-default-config35 10:44:28,222 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config35: ChangeServersVotingStatus succeeded for shard default 10:44:28,374 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config35#1919483149] 10:44:28,386 AM [main] [INFO] ShardManagerTest - testShutDown starting 10:44:28,389 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config36 10:44:28,390 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config36 10:44:28,390 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config36: Creating local shard: member-1-shard-shard2-config36 10:44:28,391 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config36: Creating local shard: member-1-shard-shard1-config36 10:44:28,401 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:28,402 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config36 10:44:28,402 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config36 10:44:28,402 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config36: Initializing shard [shard1] 10:44:28,402 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config36 is initialized 10:44:28,402 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config36: Initializing shard [shard2] 10:44:28,402 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config36 is initialized 10:44:28,402 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config36: Issuing gracefulStop to shard member-1-shard-shard2-config36 10:44:28,403 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config36: Issuing gracefulStop to shard member-1-shard-shard1-config36 10:44:28,403 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Shutting down ShardManager shard-manager-config36 - waiting on 2 shards 10:44:28,954 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-shard1-config36#2099171083] 10:44:28,955 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-shard2-config36#351048328] 10:44:28,956 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: All shards shutdown - sending PoisonPill to self 10:44:28,956 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config36 10:44:28,957 AM [main] [INFO] ShardManagerTest - testShutDown ending 10:44:29,58 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$t#1092098626] 10:44:29,72 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShardBeforeMemberUp starting 10:44:29,75 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config37 10:44:29,77 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config37 10:44:29,78 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config37: Creating local shard: member-1-shard-default-config37 10:44:29,87 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:29,87 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config37 10:44:29,87 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config37: Initializing shard [default] 10:44:29,87 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config37 is initialized 10:44:29,87 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config37: Received role changed for member-1-shard-default-config37 from Candidate to Follower 10:44:29,87 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManagerTest$TestShardManager - Received unhandled message LeaderStateChanged [memberId=member-1-shard-default-config37, leaderId=member-2-shard-default-config37, leaderPayloadVersion=13] 10:44:29,89 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config37: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 10:44:29,89 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config37: Returning NoShardLeaderException for shard default 10:44:29,89 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShardBeforeMemberUp ending 10:44:29,191 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$u#-621954961] 10:44:29,202 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config37 10:44:29,205 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithNoRoleShard starting 10:44:29,207 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config38 10:44:29,209 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config38 10:44:29,209 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: Creating local shard: member-1-shard-default-config38 10:44:29,219 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:29,219 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config38 10:44:29,219 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config38: Initializing shard [default] 10:44:29,219 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config38 is initialized 10:44:29,219 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config38: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:29,219 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config38: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config38, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 10:44:30,431 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config38: Received ShardNotInitializedTimeout message for shard default 10:44:30,431 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config38: Returning NoShardLeaderException for shard default 10:44:30,432 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithNoRoleShard ending 10:44:30,533 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$v#-187696307] 10:44:30,544 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config38 10:44:30,547 AM [main] [INFO] ShardManagerTest - testShardPersistenceWithRestoredData starting 10:44:30,551 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config39 10:44:30,553 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config39: onSnapshotOffer: ShardManagerSnapshot [ShardList = [default, astronauts] ] 10:44:30,553 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config39: adding shard astronauts 10:44:30,554 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config39: removing shard people 10:44:30,554 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config39 10:44:30,554 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config39: Creating local shard: member-1-shard-default-config39 10:44:30,555 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config39: Creating local shard: member-1-shard-astronauts-config39 10:44:30,566 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : people 10:44:30,566 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Local shard people not found - shards present: [default, astronauts] 10:44:30,566 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : default 10:44:30,566 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Returning NotInitializedException for shard default 10:44:30,566 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : astronauts 10:44:30,566 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Returning NotInitializedException for shard astronauts 10:44:30,567 AM [main] [INFO] ShardManagerTest - testShardPersistenceWithRestoredData ending 10:44:30,668 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$i] 10:44:30,680 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config39 10:44:30,683 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShard starting 10:44:30,686 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config40 10:44:30,688 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config40 10:44:30,689 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config40: Creating local shard: member-1-shard-default-config40 10:44:30,701 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config40 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config40: Initializing shard [default] 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config40 is initialized 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config40: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config40 with address akka://test/user/shardmanager-config40/member-2-shard-default-config40 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config40 with address akka://test/user/shardmanager-config40/member-2-shard-default-config40 to akka://test/user/member-1-shard-default-config 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config40: Received role changed for member-1-shard-default-config40 from Candidate to Follower 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config40: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config40, leaderId=member-2-shard-default-config40, leaderPayloadVersion=12] 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config40: All Shards are ready - data store config40 is ready 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config40: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 10:44:30,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config40: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config40/member-2-shard-default-config40, primaryVersion=12] 10:44:30,702 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShard ending 10:44:30,804 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$w#1914409305] 10:44:30,815 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config40 10:44:30,818 AM [main] [INFO] ShardManagerTest - testShardAvailabilityOnChangeOfMemberReachability starting 10:44:30,825 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 10:44:30,834 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [941715973876567231] 10:44:30,834 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 10:44:30,842 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 10:44:30,842 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 10:44:30,842 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:30,842 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:30,856 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 10:44:30,857 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 10:44:30,860 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 10:44:30,860 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config41] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:30,861 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config41 10:44:30,890 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config41 10:44:30,890 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Creating local shard: member-1-shard-default-config41 10:44:30,891 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 10:44:30,891 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 10:44:30,900 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 10:44:30,914 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-7292656492376127507] 10:44:30,914 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 10:44:30,918 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:30,919 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 10:44:30,926 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:30,926 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:30,936 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Starting ShardManager shard-manager-config41 10:44:30,938 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config41] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:30,943 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:30,943 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config41 10:44:30,943 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 10:44:30,943 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 10:44:30,943 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: updated peer member-1-shard-default-config41 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 on shard member-1-shard-default-config41 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#1884227727] 10:44:30,943 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Initializing shard [default] 10:44:30,943 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is initialized 10:44:30,943 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config41, leaderId=member-2-shard-default-config41, leaderPayloadVersion=13] 10:44:30,944 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received role changed for member-1-shard-default-config41 from Candidate to Follower 10:44:30,957 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Recovery complete : shard-manager-config41 10:44:30,957 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config41: Creating local shard: member-2-shard-default-config41 10:44:30,959 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:30,959 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config41 10:44:30,959 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config41: Initializing shard [default] 10:44:30,959 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-2-shard-default-config41 is initialized 10:44:30,959 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config41: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config41, leaderId=member-2-shard-default-config41, leaderPayloadVersion=13] 10:44:30,959 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config41: Received role changed for member-2-shard-default-config41 from Candidate to Leader 10:44:30,959 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 10:44:31,57 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 10:44:31,63 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 10:44:31,63 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 10:44:31,63 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 10:44:31,63 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 to akka://cluster-test/user/member-2-shard-default-config 10:44:31,63 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 10:44:31,854 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 10:44:31,856 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:31,856 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 10:44:31,856 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 10:44:31,856 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 10:44:31,857 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:31,857 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=13] 10:44:31,858 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 10:44:31,858 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as unavailable. 10:44:31,860 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received MemberRemoved: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 10:44:31,860 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:31,860 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config41, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config41, activeMember=true] 10:44:31,952 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:31,952 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 10:44:31,952 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-2-shard-default-config 10:44:31,953 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 10:44:33,72 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Received ShardNotInitializedTimeout message for shard default 10:44:33,72 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Returning NoShardLeaderException for shard default 10:44:33,73 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Received ReachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 10:44:33,73 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 10:44:33,73 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 10:44:33,74 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as available. 10:44:33,74 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:33,74 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=13] 10:44:33,75 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 10:44:33,75 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 10:44:33,75 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 10:44:33,75 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 10:44:33,75 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 10:44:33,75 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as unavailable. 10:44:33,75 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:33,75 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config41: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config41, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config41, activeMember=true] 10:44:33,76 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Received ReachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 10:44:33,76 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 10:44:33,76 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 10:44:33,76 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is initialized - notifying 1 OnShardInitialized callbacks 10:44:33,76 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as available. 10:44:33,76 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is ready - notifying 1 OnShardInitialized callbacks 10:44:33,76 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=13] 10:44:33,76 AM [main] [INFO] ShardManagerTest - testShardAvailabilityOnChangeOfMemberReachability ending 10:44:33,178 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config41 10:44:33,179 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:33,179 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:33,193 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:33,193 AM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:33,194 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 10:44:33,205 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Stopping ShardManager shard-manager-config41 10:44:33,205 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:33,205 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:34,232 AM [cluster-test-akka.actor.default-dispatcher-13] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 10:44:34.234] [cluster-test-akka.actor.internal-dispatcher-2] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:34,257 AM [main] [INFO] ShardManagerTest - testAddShardReplica starting 10:44:34,271 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 10:44:34,281 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [-8709687316969192312] 10:44:34,281 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 10:44:34,284 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 10:44:34,284 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 10:44:34,296 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:34,296 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:34,310 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 10:44:34,313 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config42] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:34,313 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 10:44:34,313 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 10:44:34,317 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shardmanager-config42 10:44:34,352 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shardmanager-config42 10:44:34,352 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-1-shard-default-config42 10:44:34,353 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 10:44:34,353 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 10:44:34,355 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 10:44:34,369 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [2367126624374382879] 10:44:34,369 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 10:44:34,372 AM [cluster-test-akka.actor.default-dispatcher-6] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:34,372 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 10:44:34,374 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 10:44:34,374 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 10:44:34,397 AM [cluster-test-akka.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config42] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 10:44:34,400 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:34,401 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config42 10:44:34,401 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 10:44:34,401 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 to akka://cluster-test/user/member-1-shard-default-config 10:44:34,401 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shardmanager-config42 10:44:34,401 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: updated peer member-1-shard-default-config42 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 on shard member-1-shard-default-config42 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#-2094644152] 10:44:34,403 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Recovery complete : shardmanager-config42 10:44:34,404 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-2-shard-default-config42 10:44:34,405 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-2-shard-astronauts-config42 10:44:34,405 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:34,405 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config42 10:44:34,405 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-2-shard-astronauts-config42 10:44:34,405 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shardmanager-config42: Initializing shard [astronauts] 10:44:34,405 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-2-shard-astronauts-config42 is initialized 10:44:34,405 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shardmanager-config42: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-astronauts-config42, leaderId=member-2-shard-astronauts-config42, leaderPayloadVersion=12] 10:44:34,405 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shardmanager-config42: Received role changed for member-2-shard-astronauts-config42 from Candidate to Leader 10:44:34,463 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 10:44:34,467 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 10:44:34,468 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 10:44:34,468 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 10:44:34,468 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 10:44:34,468 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-astronauts-config42 10:44:34,468 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-astronauts-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 10:44:35,314 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 10:44:35,316 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:35,316 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 10:44:35,316 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 to akka://cluster-test/user/member-1-shard-default-config 10:44:35,319 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: onAddShardReplica: AddShardReplica[ShardName=astronauts] 10:44:35,320 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=true] 10:44:35,320 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: findPrimary for astronauts forwarding to remote ShardManager akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42, visitedAddresses: [akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42] 10:44:35,323 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shardmanager-config42: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.RemoteFindPrimary [shardName=astronauts, waitUntilReady=true] 10:44:35,323 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shardmanager-config42: Found primary for astronauts: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config, primaryVersion=12] 10:44:35,328 AM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shardmanager-config42: Sending AddServer message to peer akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config for shard member-1-shard-astronauts-config42 10:44:35,334 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: Received AddServerReply{status=OK, leaderHint=member-2-shard-astronauts-config42} for shard astronauts from leader akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config 10:44:35,334 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: Leader shard successfully added the replica shard astronauts 10:44:35,335 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-astronauts-config42 10:44:35,335 AM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: persisting the shard list [default, astronauts] 10:44:35,336 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 10:44:35,336 AM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42: Successfully deleted prior snapshots 10:44:35,371 AM [main] [INFO] ShardManagerTest - testAddShardReplica ending 10:44:35,400 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 10:44:35,400 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 10:44:35,400 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 10:44:35,400 AM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-astronauts-config42 10:44:35,400 AM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-astronauts-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 10:44:35,477 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shardmanager-config42 10:44:35,479 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:35,479 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:35,494 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:35,494 AM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 10:44:35,495 AM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 10:44:35,521 AM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shardmanager-config42 10:44:35,522 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:35,522 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:36,551 AM [cluster-test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 10:44:36.552] [cluster-test-akka.actor.internal-dispatcher-4] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 10:44:36,574 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config43 10:44:36,576 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config43 10:44:36,576 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config43: Creating local shard: member-1-shard-default-config43 10:44:36,587 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:36,587 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config43 10:44:36,587 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config43: Initializing shard [default] 10:44:36,587 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config43 is initialized 10:44:36,587 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config43: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config43, leaderId=member-1-shard-default-config43, leaderPayloadVersion=13] 10:44:36,587 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config43: Received role changed for member-1-shard-default-config43 from Candidate to Leader 10:44:36,587 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config43: All Shards are ready - data store config43 is ready 10:44:36,587 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config43: onRemoveShardReplica: RemoveShardReplica [shardName=default, memberName=MemberName{name=member-1}] 10:44:36,588 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config43: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:36,588 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config43: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config43#-1506383397, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$LtBPDnsk@69a04685] 10:44:36,589 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config43: Sending RemoveServer message to peer akka://test@127.0.0.1:2565/user/member-1-shard-default-config43#-1506383397 for shard member-1-shard-default-config43 10:44:36,589 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Received RemoveServerReply{status=OK} for shard default 10:44:36,589 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Leader shard successfully removed the replica shard default 10:44:36,739 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config43#-1506383397] 10:44:36,753 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithIsolatedLeaderShard starting 10:44:36,756 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config44 10:44:36,759 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config44 10:44:36,759 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config44: Creating local shard: member-1-shard-default-config44 10:44:36,766 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:36,766 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config44 10:44:36,766 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: Initializing shard [default] 10:44:36,766 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config44 is initialized 10:44:36,767 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config44: Received role changed for member-1-shard-default-config44 from null to IsolatedLeader 10:44:36,767 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 10:44:36,767 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config44, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=IsolatedLeader, leaderId=null, activeMember=true] 10:44:37,981 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: Received ShardNotInitializedTimeout message for shard default 10:44:37,981 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: Returning NoShardLeaderException for shard default 10:44:37,982 AM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithIsolatedLeaderShard ending 10:44:38,83 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$y#229901676] 10:44:38,95 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config44 10:44:38,102 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config45 10:44:38,104 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config45 10:44:38,104 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config45: Creating local shard: member-1-shard-default-config45 10:44:38,113 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:38,113 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config45 10:44:38,113 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config45: findLocalShard : non-existent 10:44:38,113 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config45: Local shard non-existent not found - shards present: [default] 10:44:38,215 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$z#456315233] 10:44:38,226 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config45 10:44:38,232 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config46 10:44:38,233 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config46 10:44:38,253 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:38,253 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config46: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-2}] 10:44:38,254 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config46: Sending RemoveServer message to peer akka://test@127.0.0.1:2565/system/testProbe-595#-111474827 for shard member-2-shard-astronauts-config46 10:44:38,265 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config46: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-3}] 10:44:38,265 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config46: A shard replica operation for astronauts is already in progress 10:44:38,373 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config47 10:44:38,374 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config47 10:44:38,374 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config47: Creating local shard: member-1-shard-default-config47 10:44:38,383 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config47: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 10:44:38,383 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config47: Returning NotInitializedException for shard default 10:44:38,485 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$A#-722797346] 10:44:38,496 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config47 10:44:38,498 AM [main] [INFO] ShardManagerTest - testServerRemovedShardActorRunning starting 10:44:38,501 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config48 10:44:38,503 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config48 10:44:38,503 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48: Creating local shard: member-1-shard-default-config48 10:44:38,503 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48: Creating local shard: member-1-shard-people-config48 10:44:38,612 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:38,612 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config48 10:44:38,612 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-people-config48 10:44:38,613 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48: Initializing shard [default] 10:44:38,613 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-default-config48 is initialized 10:44:38,613 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48: findLocalShard : people 10:44:38,613 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-people-config48, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 10:44:38,616 AM [test-shard-dispatcher-17] [INFO] Shard - Shard created : member-1-shard-people-config48, persistent : true 10:44:38,619 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$j/member-1-shard-people-config48/shard-member-1:snapshot-read#-464086031] 10:44:38,619 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$j/member-1-shard-people-config48/member-1-shard-people-config48-notifier#-657004342 created and ready for shard:member-1-shard-people-config48 10:44:38,619 AM [test-shard-dispatcher-17] [INFO] Shard - Starting recovery for member-1-shard-people-config48 with journal batch size 1 10:44:38,621 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-people-config48: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 10:44:38,621 AM [test-shard-dispatcher-17] [INFO] Shard - member-1-shard-people-config48: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 10:44:38,621 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config48: Initializing shard [people] 10:44:38,621 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-people-config48 is initialized 10:44:38,621 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-people-config48 is initialized - notifying 1 OnShardInitialized callbacks 10:44:38,621 AM [test-shard-dispatcher-17] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-people-config48: onLeadershipChange, isLeader: false, hasLeader : false 10:44:38,621 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-people-config48: onStateChanged: Closing all transaction chains because shard member-1-shard-people-config48 is no longer the leader 10:44:38,621 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-people-config48 , received role change from null to Follower 10:44:38,621 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48: findLocalShard : default 10:44:38,621 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-people-config48 , registered listener akka://test/user/$$j 10:44:38,622 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config48: Received role changed for member-1-shard-people-config48 from null to Follower 10:44:38,622 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48 : Sending Shutdown to Shard actor Actor[akka://test/user/member-1-shard-default-config48#2100611012] with 10000 ms timeout 10:44:38,624 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48 : Local Shard replica for shard default has been removed 10:44:38,624 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48: persisting the shard list [people] 10:44:38,625 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 10:44:38,625 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48: Successfully deleted prior snapshots 10:44:38,625 AM [main] [INFO] ShardManagerTest - testServerRemovedShardActorRunning ending 10:44:38,727 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config48#2100611012] 10:44:38,738 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$j] 10:44:38,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config48 : Successfully shut down Shard actor Actor[akka://test/user/member-1-shard-default-config48#2100611012] 10:44:38,738 AM [test-shard-dispatcher-17] [INFO] Shard - Stopping Shard member-1-shard-people-config48 10:44:38,738 AM [test-shard-dispatcher-17] [DEBUG] MessageAssembler - member-1-shard-people-config48 (Follower): Closing 10:44:38,738 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-people-config48: Aborting 0 pending queued transactions 10:44:38,739 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config48 10:44:38,743 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config49 10:44:38,744 AM [test-akka.actor.default-dispatcher-18] [INFO] ShardManager - Recovery complete : shard-manager-config49 10:44:38,744 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config49: Creating local shard: member-1-shard-default-config49 10:44:38,745 AM [main] [INFO] ShardManager - shard-manager-config49: Received role changed for member-1-shard-default-config49 from null to Follower 10:44:38,746 AM [main] [INFO] ShardManager - shard-manager-config49: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config49, leaderId=member-2-shard-default-config49, leaderPayloadVersion=13] 10:44:38,756 AM [main] [INFO] ShardManager - shard-manager-config49: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 10:44:38,756 AM [main] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config49 with address akka://test/user/shardmanager-config49/member-2-shard-default-config49 10:44:38,756 AM [main] [INFO] ShardManager - shard-manager-config49: All Shards are ready - data store config49 is ready 10:44:38,858 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$k] 10:44:38,869 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config49 10:44:38,871 AM [main] [INFO] ShardManagerTest - testRestoreFromSnapshot starting 10:44:38,875 AM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config50 10:44:38,878 AM [test-akka.actor.default-dispatcher-18] [INFO] ShardManager - Recovery complete : shard-manager-config50 10:44:38,878 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config50: Restoring from ShardManagerSnapshot: ShardManagerSnapshot [ShardList = [shard1, shard2, astronauts] ] 10:44:38,878 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config50: onSnapshotOffer: ShardManagerSnapshot [ShardList = [shard1, shard2, astronauts] ] 10:44:38,878 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config50: adding shard shard1 10:44:38,878 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config50: adding shard shard2 10:44:38,878 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config50: adding shard astronauts 10:44:38,878 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-shard2-config50 10:44:38,878 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-astronauts-config50 10:44:38,878 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-shard1-config50 10:44:38,975 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:38,975 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config50 10:44:38,976 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-astronauts-config50 10:44:38,976 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config50 10:44:38,977 AM [test-shard-dispatcher-17] [INFO] Shard - Shard created : member-1-shard-shard2-config50, persistent : true 10:44:38,977 AM [test-shard-dispatcher-19] [INFO] Shard - Shard created : member-1-shard-astronauts-config50, persistent : true 10:44:38,978 AM [test-akka.actor.default-dispatcher-15] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-shard2-config50/member-1-shard-shard2-config50-notifier#-1369957301 created and ready for shard:member-1-shard-shard2-config50 10:44:38,978 AM [test-shard-dispatcher-17] [INFO] Shard - Starting recovery for member-1-shard-shard2-config50 with journal batch size 1 10:44:38,978 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-shard2-config50/shard-member-1:snapshot-read#-1516037598] 10:44:38,979 AM [test-shard-dispatcher-19] [INFO] Shard - Starting recovery for member-1-shard-astronauts-config50 with journal batch size 1 10:44:38,979 AM [test-shard-dispatcher-17] [INFO] Shard - Shard created : member-1-shard-shard1-config50, persistent : true 10:44:38,979 AM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-astronauts-config50/shard-member-1:snapshot-read#1536980663] 10:44:38,979 AM [test-akka.actor.default-dispatcher-18] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-astronauts-config50/member-1-shard-astronauts-config50-notifier#519603840 created and ready for shard:member-1-shard-astronauts-config50 10:44:38,979 AM [test-akka.actor.default-dispatcher-18] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-shard1-config50/member-1-shard-shard1-config50-notifier#2024393553 created and ready for shard:member-1-shard-shard1-config50 10:44:38,979 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-shard1-config50/shard-member-1:snapshot-read#-2008172776] 10:44:38,980 AM [test-shard-dispatcher-17] [INFO] Shard - Starting recovery for member-1-shard-shard1-config50 with journal batch size 1 10:44:38,981 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 10:44:38,981 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 10:44:38,982 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [astronauts] 10:44:38,982 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config50 is initialized 10:44:38,982 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: false, hasLeader : false 10:44:38,982 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config50 is no longer the leader 10:44:38,982 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from null to Follower 10:44:38,983 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50 (Follower): Received TimeoutNow - switching to Candidate 10:44:38,984 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 10:44:38,984 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard2-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 10:44:38,984 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-shard1-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 10:44:38,984 AM [test-shard-dispatcher-17] [INFO] Shard - member-1-shard-shard2-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 10:44:38,984 AM [test-shard-dispatcher-17] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: false, hasLeader : false 10:44:38,984 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard2-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config50 is no longer the leader 10:44:38,984 AM [test-shard-dispatcher-22] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: false, hasLeader : false 10:44:38,984 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config50 is no longer the leader 10:44:38,984 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard2-config50 (Follower): Received TimeoutNow - switching to Candidate 10:44:38,984 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50 (Follower): Received TimeoutNow - switching to Candidate 10:44:38,987 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50 (Candidate): Election: Candidate has following voting peers: [] 10:44:38,987 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard2-config50 (Candidate): Election: Candidate has following voting peers: [] 10:44:38,987 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard2-config50: Set currentTerm=1, votedFor=member-1-shard-shard2-config50 10:44:38,987 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: Set currentTerm=1, votedFor=member-1-shard-astronauts-config50 10:44:38,988 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50 (Candidate): Election: Candidate has following voting peers: [] 10:44:38,988 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50: Set currentTerm=1, votedFor=member-1-shard-shard1-config50 10:44:38,991 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-shard1-config50 (Candidate): Starting new election term 1 10:44:38,991 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-shard1-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 10:44:38,991 AM [test-shard-dispatcher-22] [DEBUG] MessageAssembler - member-1-shard-shard1-config50 (Follower): Closing 10:44:38,991 AM [test-shard-dispatcher-22] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: false, hasLeader : false 10:44:38,991 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config50 is no longer the leader 10:44:38,991 AM [test-shard-dispatcher-17] [INFO] Shard - member-1-shard-shard2-config50 (Candidate): Starting new election term 1 10:44:38,991 AM [test-shard-dispatcher-17] [INFO] Shard - member-1-shard-shard2-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 10:44:38,991 AM [test-shard-dispatcher-17] [DEBUG] MessageAssembler - member-1-shard-shard2-config50 (Follower): Closing 10:44:38,991 AM [test-shard-dispatcher-17] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: false, hasLeader : false 10:44:38,991 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard2-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config50 is no longer the leader 10:44:38,992 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config50 (Candidate): Starting new election term 1 10:44:38,992 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 10:44:38,992 AM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-astronauts-config50 (Follower): Closing 10:44:38,992 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: false, hasLeader : false 10:44:38,992 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config50 is no longer the leader 10:44:38,992 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [shard1] 10:44:38,992 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config50 is initialized 10:44:38,992 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [shard2] 10:44:38,992 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config50 is initialized 10:44:38,992 AM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from null to Follower 10:44:38,992 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : shard1 10:44:38,992 AM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from Follower to Candidate 10:44:38,992 AM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from null to Follower 10:44:38,992 AM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from Follower to Candidate 10:44:38,992 AM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from Follower to Candidate 10:44:38,993 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : shard2 10:44:38,993 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : astronauts 10:44:38,993 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config50: onGetSnapshot 10:44:38,996 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Expecting 3 shard snapshot replies 10:44:39,6 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard1-config50 (Candidate): Received ElectionTimeout 10:44:39,10 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config50 (Candidate): Received ElectionTimeout 10:44:39,14 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50 (Candidate): Received ElectionTimeout 10:44:39,21 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard1-config50 (Leader): Election: Leader has following peers: [] 10:44:39,21 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50 (Leader): Election: Leader has following peers: [] 10:44:39,21 AM [test-shard-dispatcher-17] [INFO] Shard - member-1-shard-shard1-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 10:44:39,21 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 10:44:39,21 AM [test-shard-dispatcher-17] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: true, hasLeader : true 10:44:39,21 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: true, hasLeader : true 10:44:39,22 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config50 (Leader): Election: Leader has following peers: [] 10:44:39,22 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-shard2-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 10:44:39,22 AM [test-shard-dispatcher-22] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: true, hasLeader : true 10:44:39,23 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard1-config50: became leader with frontend state for [] 10:44:39,23 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config50: became leader with frontend state for [] 10:44:39,23 AM [test-akka.actor.default-dispatcher-23] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from Candidate to Leader 10:44:39,23 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from Candidate to Leader 10:44:39,24 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: became leader with frontend state for [] 10:44:39,24 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from Candidate to Leader 10:44:39,28 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard1-config50: onGetSnapshot 10:44:39,28 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: onGetSnapshot 10:44:39,28 AM [test-akka.actor.default-dispatcher-23] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , registered listener akka://test/user/$$l 10:44:39,28 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard1-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 10:44:39,28 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , registered listener akka://test/user/$$l 10:44:39,28 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-astronauts-config50, leaderId=member-1-shard-astronauts-config50, leaderPayloadVersion=13] 10:44:39,28 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 10:44:39,28 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-astronauts-config50 from Candidate to Leader 10:44:39,28 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config50: onGetSnapshot 10:44:39,28 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard1-config50, leaderId=member-1-shard-shard1-config50, leaderPayloadVersion=13] 10:44:39,28 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 10:44:39,28 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-shard1-config50 from Candidate to Leader 10:44:39,28 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , registered listener akka://test/user/$$l 10:44:39,28 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard2-config50, leaderId=member-1-shard-shard2-config50, leaderPayloadVersion=13] 10:44:39,28 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-shard2-config50 from Candidate to Leader 10:44:39,28 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config50: All Shards are ready - data store config50 is ready 10:44:39,35 AM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-shard1-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 10:44:39,35 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 10:44:39,35 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 10:44:39,48 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-astronauts-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@3328ce7, electionTerm=1, electionVotedFor=member-1-shard-astronauts-config50, ServerConfigPayload=null]] 10:44:39,50 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-shard1-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@1179ed5a, electionTerm=1, electionVotedFor=member-1-shard-shard1-config50, ServerConfigPayload=null]] 10:44:39,50 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-shard2-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@6bb1e26, electionTerm=1, electionVotedFor=member-1-shard-shard2-config50, ServerConfigPayload=null]] 10:44:39,50 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: All shard snapshots received 10:44:39,52 AM [main] [INFO] ShardManagerTest - testRestoreFromSnapshot ending 10:44:39,154 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$l] 10:44:39,166 AM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-astronauts-config50 10:44:39,166 AM [test-shard-dispatcher-19] [DEBUG] MessageSlicer - member-1-shard-astronauts-config50 (Leader)_slicer-id-215: Closing 10:44:39,166 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: Aborting 0 pending queued transactions 10:44:39,166 AM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-shard1-config50 10:44:39,166 AM [test-shard-dispatcher-19] [DEBUG] MessageSlicer - member-1-shard-shard1-config50 (Leader)_slicer-id-214: Closing 10:44:39,167 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config50: Aborting 0 pending queued transactions 10:44:39,167 AM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-shard2-config50 10:44:39,167 AM [test-shard-dispatcher-19] [DEBUG] MessageSlicer - member-1-shard-shard2-config50 (Leader)_slicer-id-216: Closing 10:44:39,167 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config50: Aborting 0 pending queued transactions 10:44:39,167 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config50 10:44:39,169 AM [main] [INFO] ShardManagerTest - testOnCreateShardWithNoInitialSchemaContext starting 10:44:39,173 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config51 10:44:39,174 AM [test-akka.actor.default-dispatcher-21] [INFO] ShardManager - Recovery complete : shard-manager-config51 10:44:39,183 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config51: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@2e4b7cca] 10:44:39,184 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config51 doCreateShard: shardId: member-1-shard-foo-config51, memberNames: [MemberName{name=member-1}], peerAddresses: {}, isActiveMember: true 10:44:39,185 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:39,185 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-foo-config51 10:44:39,185 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config51: findLocalShard : foo 10:44:39,185 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config51: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config51, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 10:44:39,186 AM [test-shard-dispatcher-22] [INFO] Shard - Shard created : member-1-shard-foo-config51, persistent : true 10:44:39,187 AM [test-shard-dispatcher-22] [INFO] Shard - Starting recovery for member-1-shard-foo-config51 with journal batch size 1 10:44:39,187 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$B/member-1-shard-foo-config51/shard-member-1:snapshot-read#1656893750] 10:44:39,188 AM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$B/member-1-shard-foo-config51/member-1-shard-foo-config51-notifier#118307439 created and ready for shard:member-1-shard-foo-config51 10:44:39,188 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-foo-config51: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 10:44:39,188 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-foo-config51: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 10:44:39,189 AM [test-shard-dispatcher-22] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: false, hasLeader : false 10:44:39,189 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-foo-config51: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config51 is no longer the leader 10:44:39,189 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-foo-config51 (Follower): Received TimeoutNow - switching to Candidate 10:44:39,189 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-foo-config51 (Candidate): Election: Candidate has following voting peers: [] 10:44:39,189 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-foo-config51: Set currentTerm=1, votedFor=member-1-shard-foo-config51 10:44:39,189 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-foo-config51 (Candidate): Starting new election term 1 10:44:39,189 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-foo-config51 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 10:44:39,189 AM [test-shard-dispatcher-22] [DEBUG] MessageAssembler - member-1-shard-foo-config51 (Follower): Closing 10:44:39,189 AM [test-shard-dispatcher-22] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: false, hasLeader : false 10:44:39,189 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-foo-config51: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config51 is no longer the leader 10:44:39,189 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config51: Initializing shard [foo] 10:44:39,189 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardInformation - Shard member-1-shard-foo-config51 is initialized 10:44:39,189 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardInformation - Shard member-1-shard-foo-config51 is initialized - notifying 1 OnShardInitialized callbacks 10:44:39,189 AM [test-akka.actor.default-dispatcher-23] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from null to Follower 10:44:39,189 AM [test-akka.actor.default-dispatcher-23] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from Follower to Candidate 10:44:39,189 AM [main] [INFO] ShardManagerTest - testOnCreateShardWithNoInitialSchemaContext ending 10:44:39,190 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-foo-config51 (Candidate): Received ElectionTimeout 10:44:39,190 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-foo-config51 (Leader): Election: Leader has following peers: [] 10:44:39,190 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-foo-config51 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 10:44:39,190 AM [test-shard-dispatcher-22] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: true, hasLeader : true 10:44:39,190 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-foo-config51: became leader with frontend state for [] 10:44:39,190 AM [test-akka.actor.default-dispatcher-23] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from Candidate to Leader 10:44:39,190 AM [test-akka.actor.default-dispatcher-23] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , registered listener akka://test/user/$B 10:44:39,191 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - shard-manager-config51: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-foo-config51, leaderId=member-1-shard-foo-config51, leaderPayloadVersion=13] 10:44:39,191 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - shard-manager-config51: Received role changed for member-1-shard-foo-config51 from Candidate to Leader 10:44:39,191 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - shard-manager-config51: All Shards are ready - data store config51 is ready 10:44:39,293 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$B#-724279234] 10:44:39,305 AM [test-shard-dispatcher-22] [INFO] Shard - Stopping Shard member-1-shard-foo-config51 10:44:39,305 AM [test-shard-dispatcher-22] [DEBUG] MessageSlicer - member-1-shard-foo-config51 (Leader)_slicer-id-218: Closing 10:44:39,305 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-foo-config51: Aborting 0 pending queued transactions 10:44:39,306 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - Stopping ShardManager shard-manager-config51 10:44:39,310 AM [main] [INFO] ShardManagerTest - testGetSnapshot starting 10:44:39,313 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - Starting ShardManager shard-manager-config52 10:44:39,314 AM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config52 10:44:39,314 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: Creating local shard: member-1-shard-shard2-config52 10:44:39,314 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config52: Creating local shard: member-1-shard-shard1-config52 10:44:39,323 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 10:44:39,324 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:39,324 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config52 10:44:39,325 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config52 10:44:39,325 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : shard1 10:44:39,325 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-shard1-config52, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 10:44:39,326 AM [test-shard-dispatcher-22] [INFO] Shard - Shard created : member-1-shard-shard2-config52, persistent : true 10:44:39,326 AM [test-shard-dispatcher-19] [INFO] Shard - Shard created : member-1-shard-shard1-config52, persistent : true 10:44:39,328 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-shard2-config52/member-1-shard-shard2-config52-notifier#-144204320 created and ready for shard:member-1-shard-shard2-config52 10:44:39,328 AM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-shard1-config52/member-1-shard-shard1-config52-notifier#-344133383 created and ready for shard:member-1-shard-shard1-config52 10:44:39,328 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-shard2-config52/shard-member-1:snapshot-read#1319732573] 10:44:39,328 AM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-shard1-config52/shard-member-1:snapshot-read#-1514529093] 10:44:39,328 AM [test-shard-dispatcher-19] [INFO] Shard - Starting recovery for member-1-shard-shard1-config52 with journal batch size 1 10:44:39,328 AM [test-shard-dispatcher-22] [INFO] Shard - Starting recovery for member-1-shard-shard2-config52 with journal batch size 1 10:44:39,337 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 10:44:39,337 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 10:44:39,337 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard1-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 10:44:39,337 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-shard2-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 10:44:39,337 AM [test-shard-dispatcher-22] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: false, hasLeader : false 10:44:39,337 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: false, hasLeader : false 10:44:39,337 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config52 is no longer the leader 10:44:39,337 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config52 is no longer the leader 10:44:39,337 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from null to Follower 10:44:39,338 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config52 (Follower): Received TimeoutNow - switching to Candidate 10:44:39,338 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52 (Follower): Received TimeoutNow - switching to Candidate 10:44:39,338 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52 (Candidate): Election: Candidate has following voting peers: [] 10:44:39,338 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config52 (Candidate): Election: Candidate has following voting peers: [] 10:44:39,338 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: Set currentTerm=1, votedFor=member-1-shard-shard1-config52 10:44:39,338 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config52: Set currentTerm=1, votedFor=member-1-shard-shard2-config52 10:44:39,338 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [shard1] 10:44:39,338 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config52 is initialized 10:44:39,338 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config52 is initialized - notifying 1 OnShardInitialized callbacks 10:44:39,338 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard1-config52 (Candidate): Starting new election term 1 10:44:39,338 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-shard2-config52 (Candidate): Starting new election term 1 10:44:39,338 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-shard2-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 10:44:39,338 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard1-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 10:44:39,338 AM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-shard1-config52 (Follower): Closing 10:44:39,338 AM [test-shard-dispatcher-22] [DEBUG] MessageAssembler - member-1-shard-shard2-config52 (Follower): Closing 10:44:39,338 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: false, hasLeader : false 10:44:39,338 AM [test-shard-dispatcher-22] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: false, hasLeader : false 10:44:39,338 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config52 is no longer the leader 10:44:39,338 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard2-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config52 is no longer the leader 10:44:39,338 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [shard2] 10:44:39,338 AM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from Follower to Candidate 10:44:39,338 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from null to Follower 10:44:39,338 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config52 is initialized 10:44:39,338 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from Follower to Candidate 10:44:39,338 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : shard2 10:44:39,338 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 10:44:39,339 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Expecting 2 shard snapshot replies 10:44:39,340 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config52 (Candidate): Received ElectionTimeout 10:44:39,340 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52 (Candidate): Received ElectionTimeout 10:44:39,340 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52 (Leader): Election: Leader has following peers: [] 10:44:39,340 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config52 (Leader): Election: Leader has following peers: [] 10:44:39,340 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard2-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 10:44:39,340 AM [test-shard-dispatcher-22] [INFO] Shard - member-1-shard-shard1-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 10:44:39,340 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: true, hasLeader : true 10:44:39,340 AM [test-shard-dispatcher-22] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: true, hasLeader : true 10:44:39,340 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52: became leader with frontend state for [] 10:44:39,340 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config52: became leader with frontend state for [] 10:44:39,340 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52: onGetSnapshot 10:44:39,340 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 10:44:39,340 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config52: onGetSnapshot 10:44:39,340 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 10:44:39,340 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-shard1-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 10:44:39,340 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 10:44:39,341 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from Candidate to Leader 10:44:39,341 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from Candidate to Leader 10:44:39,341 AM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , registered listener akka://test/user/$$m 10:44:39,341 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , registered listener akka://test/user/$$m 10:44:39,341 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard2-config52, leaderId=member-1-shard-shard2-config52, leaderPayloadVersion=13] 10:44:39,341 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard1-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@c32e49c, electionTerm=1, electionVotedFor=member-1-shard-shard1-config52, ServerConfigPayload=null]] 10:44:39,341 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-shard2-config52 from Candidate to Leader 10:44:39,341 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard1-config52, leaderId=member-1-shard-shard1-config52, leaderPayloadVersion=13] 10:44:39,341 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard2-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@21e4972b, electionTerm=1, electionVotedFor=member-1-shard-shard2-config52, ServerConfigPayload=null]] 10:44:39,341 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-shard1-config52 from Candidate to Leader 10:44:39,341 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: All shard snapshots received 10:44:39,341 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config52: All Shards are ready - data store config52 is ready 10:44:39,353 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config52: onAddShardReplica: AddShardReplica[ShardName=astronauts] 10:44:39,354 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config52: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-609#1108100969 for shard member-1-shard-astronauts-config52 10:44:39,355 AM [test-shard-dispatcher-19] [INFO] Shard - Shard created : member-1-shard-astronauts-config52, persistent : true 10:44:39,355 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config52: Received AddServerReply{status=OK, leaderHint=} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-609#1108100969 10:44:39,355 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config52: Leader shard successfully added the replica shard astronauts 10:44:39,355 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-astronauts-config52 10:44:39,355 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config52: persisting the shard list [shard2, astronauts, shard1] 10:44:39,355 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : astronauts 10:44:39,355 AM [test-akka.actor.default-dispatcher-18] [DEBUG] ShardManager - shard-manager-config52: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-astronauts-config52, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 10:44:39,356 AM [test-shard-dispatcher-19] [INFO] Shard - Starting recovery for member-1-shard-astronauts-config52 with journal batch size 1 10:44:39,356 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 10:44:39,356 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-astronauts-config52/shard-member-1:snapshot-read#1127741639] 10:44:39,356 AM [test-akka.actor.default-dispatcher-18] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-astronauts-config52/member-1-shard-astronauts-config52-notifier#297268937 created and ready for shard:member-1-shard-astronauts-config52 10:44:39,357 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config52: Successfully deleted prior snapshots 10:44:39,358 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 10:44:39,358 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 10:44:39,358 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 10:44:39,358 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 10:44:39,358 AM [test-shard-dispatcher-19] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 10:44:39,358 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: RaftPolicy used with prev.config org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy, RaftPolicy used with newConfig null 10:44:39,358 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: Re-initializing to Follower with previous leaderId null 10:44:39,358 AM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-astronauts-config52 (Follower): Closing 10:44:39,358 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 10:44:39,359 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 10:44:39,359 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52 (Follower): Received TimeoutNow - switching to Candidate 10:44:39,359 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52 (Candidate): Election: Candidate has following voting peers: [] 10:44:39,359 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: Set currentTerm=1, votedFor=member-1-shard-astronauts-config52 10:44:39,359 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config52 (Candidate): Starting new election term 1 10:44:39,359 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 10:44:39,359 AM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-astronauts-config52 (Follower): Closing 10:44:39,359 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 10:44:39,359 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 10:44:39,359 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [astronauts] 10:44:39,359 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config52 is initialized 10:44:39,359 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config52 is initialized - notifying 1 OnShardInitialized callbacks 10:44:39,359 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from null to Follower 10:44:39,359 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from Follower to Candidate 10:44:39,359 AM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 10:44:39,360 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52 (Candidate): Received ElectionTimeout 10:44:39,360 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52 (Leader): Election: Leader has following peers: [] 10:44:39,360 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 10:44:39,360 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: true, hasLeader : true 10:44:39,360 AM [test-akka.actor.default-dispatcher-21] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Expecting 3 shard snapshot replies 10:44:39,360 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: became leader with frontend state for [] 10:44:39,361 AM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from Candidate to Leader 10:44:39,361 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onGetSnapshot 10:44:39,361 AM [test-akka.actor.default-dispatcher-21] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , registered listener akka://test/user/$$m 10:44:39,361 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 10:44:39,361 AM [test-akka.actor.default-dispatcher-21] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-astronauts-config52, leaderId=member-1-shard-astronauts-config52, leaderPayloadVersion=13] 10:44:39,361 AM [test-akka.actor.default-dispatcher-21] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-astronauts-config52 from Candidate to Leader 10:44:39,361 AM [test-akka.actor.default-dispatcher-21] [INFO] ShardManager - shard-manager-config52: All Shards are ready - data store config52 is ready 10:44:39,361 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 10:44:39,361 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: onGetSnapshot 10:44:39,361 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: onGetSnapshot 10:44:39,361 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 10:44:39,361 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 10:44:39,361 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 10:44:39,361 AM [test-shard-dispatcher-20] [DEBUG] Shard - member-1-shard-shard2-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 10:44:39,361 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-astronauts-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@188d6a47, electionTerm=1, electionVotedFor=member-1-shard-astronauts-config52, ServerConfigPayload=null]] 10:44:39,362 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard1-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@7e9a0afa, electionTerm=1, electionVotedFor=member-1-shard-shard1-config52, ServerConfigPayload=null]] 10:44:39,362 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard2-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@325a1880, electionTerm=1, electionVotedFor=member-1-shard-shard2-config52, ServerConfigPayload=null]] 10:44:39,362 AM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: All shard snapshots received 10:44:39,363 AM [main] [INFO] ShardManagerTest - testGetSnapshot ending 10:44:39,466 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$m] 10:44:39,477 AM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-shard1-config52 10:44:39,477 AM [test-shard-dispatcher-22] [INFO] Shard - Stopping Shard member-1-shard-astronauts-config52 10:44:39,477 AM [test-shard-dispatcher-19] [DEBUG] MessageSlicer - member-1-shard-shard1-config52 (Leader)_slicer-id-222: Closing 10:44:39,477 AM [test-shard-dispatcher-22] [DEBUG] MessageSlicer - member-1-shard-astronauts-config52 (Leader)_slicer-id-224: Closing 10:44:39,478 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: Aborting 0 pending queued transactions 10:44:39,478 AM [test-shard-dispatcher-22] [DEBUG] Shard - member-1-shard-astronauts-config52: Aborting 0 pending queued transactions 10:44:39,478 AM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-shard2-config52 10:44:39,478 AM [test-shard-dispatcher-19] [DEBUG] MessageSlicer - member-1-shard-shard2-config52 (Leader)_slicer-id-221: Closing 10:44:39,478 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52: Aborting 0 pending queued transactions 10:44:39,478 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - Stopping ShardManager shard-manager-config52 10:44:39,480 AM [main] [INFO] ShardManagerTest - testWhenMultipleShardsPresentSyncStatusMustBeTrueForAllShards starting 10:44:39,484 AM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config53 10:44:39,485 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - Recovery complete : shard-manager-config53 10:44:39,485 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config53: Creating local shard: member-1-shard-default-config53 10:44:39,485 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config53: Creating local shard: member-1-shard-astronauts-config53 10:44:39,486 AM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-default-config53 from Follower to Leader 10:44:39,486 AM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-astronauts-config53 from Follower to Leader 10:44:39,486 AM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-astronauts-config53 from Leader to Follower 10:44:39,486 AM [main] [INFO] ShardManager - shard-manager-config53 Received follower initial sync status for member-1-shard-astronauts-config53 status sync done true 10:44:39,486 AM [main] [INFO] ShardManagerTest - testWhenMultipleShardsPresentSyncStatusMustBeTrueForAllShards ending 10:44:39,588 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$n] 10:44:39,599 AM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config53 10:44:39,601 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithFindPrimaryTimeout starting 10:44:39,604 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config54 10:44:39,605 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - Recovery complete : shard-manager-config54 10:44:39,614 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:39,614 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config54: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://non-existent@127.0.0.1:5 10:44:39,614 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config54: All Shards are ready - data store config54 is ready 10:44:39,614 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config54: onAddShardReplica: AddShardReplica[ShardName=astronauts] 10:44:39,614 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config54: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=true] 10:44:39,614 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config54: findPrimary for astronauts forwarding to remote ShardManager akka://non-existent@127.0.0.1:5/user/shardmanager-config54, visitedAddresses: [akka://test@127.0.0.1:2550/user/shardmanager-config54] 10:44:39,637 AM [test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://non-existent@127.0.0.1:5], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1/:5,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused 10:44:39,637 AM [test-akka.actor.default-dispatcher-5] [WARN] Materializer - [outbound connection to [akka://non-existent@127.0.0.1:5], control stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1/:5,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused 10:44:39,830 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config54: Received failure from FindPrimary for shard astronauts akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://test/user/shardmanager-config54#-689270582]] after [200 ms]. Message of type [org.opendaylight.controller.cluster.datastore.messages.FindPrimary]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. 10:44:39,831 AM [main] [INFO] ShardManagerTest - testAddShardReplicaWithFindPrimaryTimeout ending 10:44:39,932 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/shardmanager-config54#-689270582] 10:44:39,943 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config54 10:44:39,946 AM [main] [INFO] ShardManagerTest - testOnCreateShard starting 10:44:39,948 AM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config55 10:44:39,950 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - Recovery complete : shard-manager-config55 10:44:39,950 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 10:44:39,959 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config55: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@4618c26] 10:44:39,959 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config55 doCreateShard: shardId: member-1-shard-foo-config55, memberNames: [MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], peerAddresses: {member-6-shard-foo-config55=null, member-5-shard-foo-config55=null}, isActiveMember: true 10:44:39,960 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config55: findLocalShard : foo 10:44:39,960 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config55: Scheduling 300000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config55, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 10:44:39,962 AM [test-shard-dispatcher-19] [INFO] Shard - Shard created : member-1-shard-foo-config55, persistent : false 10:44:39,965 AM [test-shard-dispatcher-19] [INFO] Shard - Starting recovery for member-1-shard-foo-config55 with journal batch size 1 10:44:39,966 AM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$C/member-1-shard-foo-config55/member-1-shard-foo-config55-notifier#-758356899 created and ready for shard:member-1-shard-foo-config55 10:44:39,966 AM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$C/member-1-shard-foo-config55/shard-member-1:snapshot-read#452229492] 10:44:39,973 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config55: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 10:44:39,973 AM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-foo-config55: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 10:44:39,973 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config55: Initializing shard [foo] 10:44:39,973 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-foo-config55 is initialized 10:44:39,973 AM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-foo-config55 is initialized - notifying 1 OnShardInitialized callbacks 10:44:39,973 AM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config55: onLeadershipChange, isLeader: false, hasLeader : false 10:44:39,973 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config55: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config55 is no longer the leader 10:44:39,974 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config55 , received role change from null to Follower 10:44:39,974 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config55: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@4618c26] 10:44:39,974 AM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config55: Shard foo already exists 10:44:39,974 AM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config55 , registered listener akka://test/user/$C 10:44:39,974 AM [main] [INFO] ShardManagerTest - testOnCreateShard ending 10:44:39,974 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - shard-manager-config55: Received role changed for member-1-shard-foo-config55 from null to Follower 10:44:40,76 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$C#275952126] 10:44:40,87 AM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-foo-config55 10:44:40,87 AM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-foo-config55 (Follower): Closing 10:44:40,87 AM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-foo-config55: Aborting 0 pending queued transactions 10:44:40,87 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config55 10:44:40,89 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - Stopping ShardManager shard-manager-config1 10:44:40,89 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config35 10:44:40,89 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - Stopping ShardManager shard-manager-config11 10:44:40,89 AM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config43 10:44:40,89 AM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config21 10:44:40,89 AM [test-akka.actor.default-dispatcher-23] [INFO] ShardManager - Stopping ShardManager shard-manager-config46 10:44:40,90 AM [test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:44:40,90 AM [test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 10:44:40,94 AM [test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/13/2024 10:44:40.096] [test-akka.actor.internal-dispatcher-3] [Cluster(akka://test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config