03:01:28,891 PM [test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 03:01:28,933 PM [test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://test@127.0.0.1:2565] with UID [6050092235552410484] 03:01:28,945 PM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Starting up, Akka version [2.6.21] ... 03:01:28,954 PM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Registered cluster JMX MBean [akka:type=Cluster] 03:01:28,954 PM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - Started up successfully 03:01:28,957 PM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:28,957 PM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@127.0.0.1:2565] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:29,819 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$a#-1127268428] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:29,835 PM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config1 03:01:29,958 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config1 03:01:29,958 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Creating local shard: member-1-shard-default-config1 03:01:29,993 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:29,993 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config1 03:01:29,994 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Initializing shard [default] 03:01:29,994 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config1 is initialized 03:01:29,995 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config1: Received role changed for member-1-shard-default-config1 from null to Follower 03:01:29,995 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: onChangeShardServersVotingStatus: ChangeShardMembersVotingStatus [shardName=default, meberVotingStatusMap={member-2=true}] 03:01:29,998 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: findLocalShard : default 03:01:30,3 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config1: Sending ChangeServersVotingStatus message ChangeServersVotingStatus [serverVotingStatusMap={member-2-shard-default-config1=true}, serversVisited=[]] to local shard akka://test/user/member-1-shard-default-config1 03:01:30,9 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config1: Received ServerChangeReply{status=NO_LEADER} from local shard akka://test/user/member-1-shard-default-config1 03:01:30,9 PM [test-akka.actor.default-dispatcher-5] [WARN] ShardManager - shard-manager-config1: ChangeServersVotingStatus failed for shard default with status NO_LEADER 03:01:30,176 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config1#856681820] 03:01:30,201 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForShardLeader starting 03:01:30,208 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$b#-1529437488] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:30,208 PM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config2 03:01:30,213 PM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config2 03:01:30,213 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config2: Creating local shard: member-1-shard-default-config2 03:01:30,225 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:30,225 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config2 03:01:30,232 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config2: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:30,246 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config2: Scheduling 10000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config2, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 03:01:30,371 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config2: Initializing shard [default] 03:01:30,371 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized 03:01:30,372 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized - notifying 1 OnShardInitialized callbacks 03:01:30,524 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config2: Received role changed for member-1-shard-default-config2 from Candidate to Leader 03:01:30,525 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is initialized - notifying 1 OnShardInitialized callbacks 03:01:30,747 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config2: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config2, leaderId=member-1-shard-default-config2, leaderPayloadVersion=13] 03:01:30,747 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config2 is ready - notifying 1 OnShardInitialized callbacks 03:01:30,750 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config2: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#212035632, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$IHbtyKwr@62ca9413] 03:01:30,751 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config2: All Shards are ready - data store config2 is ready 03:01:30,951 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForShardLeader ending 03:01:31,54 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$b#-1529437488] 03:01:31,67 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config2 03:01:31,86 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config3 03:01:31,89 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/$$a] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:31,92 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config3 03:01:31,92 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config3: Creating local shard: member-1-shard-default-config3 03:01:31,94 PM [main] [INFO] ShardManager - shard-manager-config3: Received role changed for member-1-shard-default-config3 from Follower to Candidate 03:01:31,94 PM [main] [INFO] ShardManager - shard-manager-config3 Received follower initial sync status for member-1-shard-default-config3 status sync done true 03:01:31,196 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$a] 03:01:31,208 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config3 03:01:31,250 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 03:01:31,299 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [3825219168269439793] 03:01:31,300 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 03:01:31,314 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:31,314 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 03:01:31,316 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:31,379 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:31,379 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config4] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:31,384 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Starting ShardManager shard-manager-config4 03:01:31,422 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 03:01:31,422 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 03:01:31,443 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - Recovery complete : shard-manager-config4 03:01:31,444 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-1-shard-default-config4 03:01:31,445 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-1-shard-astronauts-config4 03:01:31,446 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 03:01:31,453 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 03:01:31,490 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 03:01:31,491 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 03:01:31,491 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 03:01:31,514 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-3440677919301978524] 03:01:31,516 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 03:01:31,526 PM [cluster-test-akka.actor.default-dispatcher-6] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:31,526 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 03:01:31,536 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:31,536 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:31,568 PM [main] [ERROR] ShardManagerTest - Mock Shard Leader Actor : TestActor[akka://cluster-test/user/member-2-shard-default-config4] 03:01:31,589 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config4 03:01:31,590 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config4] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:31,593 PM [main] [ERROR] ShardManagerTest - Forwarding actor : Actor[akka://cluster-test/user/shardmanager-config4/member-2-shard-default-config4#1171130567] 03:01:31,611 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config4 03:01:31,612 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config4: Creating local shard: member-2-shard-default-config4 03:01:31,615 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:31,615 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config4 03:01:31,616 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 03:01:31,616 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 to akka://cluster-test/user/member-1-shard-default-config 03:01:31,617 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: updated peer member-1-shard-default-config4 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 on shard member-1-shard-default-config4 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#313486896] 03:01:31,617 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-1-shard-astronauts-config4 03:01:31,617 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 03:01:31,617 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 to akka://cluster-test/user/member-1-shard-default-config 03:01:31,617 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: updated peer member-1-shard-astronauts-config4 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-astronauts-config4 on shard member-1-shard-astronauts-config4 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#313486896] 03:01:31,618 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: Initializing shard [default] 03:01:31,618 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-1-shard-default-config4 is initialized 03:01:31,618 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config4: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config4, leaderId=member-2-shard-default-config4, leaderPayloadVersion=12] 03:01:31,618 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config4: Received role changed for member-1-shard-default-config4 from Candidate to Follower 03:01:31,624 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:31,624 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config4 03:01:31,624 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config4: Initializing shard [default] 03:01:31,624 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-2-shard-default-config4 is initialized 03:01:31,625 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config4: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config4, leaderId=member-2-shard-default-config4, leaderPayloadVersion=12] 03:01:31,625 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config4: Received role changed for member-2-shard-default-config4 from Candidate to Leader 03:01:31,625 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 03:01:32,81 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 03:01:32,162 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 03:01:32,162 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 03:01:32,162 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 03:01:32,162 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config4 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config4/member-1-shard-default-config4 to akka://cluster-test/user/member-2-shard-default-config4 03:01:32,162 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config4: All Shards are ready - data store config4 is ready 03:01:32,335 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 03:01:32,335 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shard-manager-config4: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:32,335 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 03:01:32,336 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 to akka://cluster-test/user/member-1-shard-default-config 03:01:32,336 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-astronauts-config4 03:01:32,336 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config4 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-astronauts-config4 to akka://cluster-test/user/member-1-shard-default-config 03:01:32,337 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: onRemoveShardReplica: RemoveShardReplica [shardName=default, memberName=MemberName{name=member-1}] 03:01:32,340 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:32,342 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4, primaryVersion=12] 03:01:32,343 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config4: Sending RemoveServer message to peer akka://cluster-test@127.0.0.1:2559/user/shardmanager-config4/member-2-shard-default-config4 for shard member-1-shard-default-config4 03:01:32,368 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: Received RemoveServerReply{status=OK, leaderHint=member-2-shard-default-config4} for shard default 03:01:32,368 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config4: Leader shard successfully removed the replica shard default 03:01:32,491 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shard-manager-config4 03:01:32,498 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:32,498 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:32,542 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:32,546 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:32,559 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 03:01:32,610 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Stopping ShardManager shard-manager-config4 03:01:32,613 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:32,613 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [WARN] [03/07/2024 15:01:33.648] [cluster-test-akka.actor.internal-dispatcher-2] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:33,675 PM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$c#-1206753059] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:33,678 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config5 03:01:33,680 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config5 03:01:33,680 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config5: Creating local shard: member-1-shard-default-config5 03:01:33,692 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:33,692 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config5 03:01:33,692 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config5: Initializing shard [default] 03:01:33,692 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-default-config5 is initialized 03:01:33,845 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$c#-1206753059] 03:01:33,858 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config5 03:01:33,870 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config6 03:01:33,874 PM [test-akka.actor.default-dispatcher-15] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/$$b] to Actor[akka://test/user/termination-monitor] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:33,876 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config6 03:01:33,876 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config6: Creating local shard: member-1-shard-default-config6 03:01:33,972 PM [main] [INFO] ShardManager - shard-manager-config6: Received role changed for member-1-shard-default-config6 from null to Follower 03:01:33,983 PM [main] [INFO] ShardManager - shard-manager-config6: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 03:01:33,983 PM [main] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config6 with address akka://test/user/shardmanager-config6/member-2-shard-default-config6 03:01:33,984 PM [main] [INFO] ShardManager - shard-manager-config6: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config6, leaderId=member-2-shard-default-config6, leaderPayloadVersion=13] 03:01:33,984 PM [main] [INFO] ShardManager - shard-manager-config6: All Shards are ready - data store config6 is ready 03:01:34,86 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$b] 03:01:34,97 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config6 03:01:34,113 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$d#-330816222] to Actor[akka://test/user/termination-monitor] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:34,114 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config7 03:01:34,117 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config7 03:01:34,125 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config7: onRemoveShardReplica: RemoveShardReplica [shardName=model-inventory, memberName=MemberName{name=member-1}] 03:01:34,126 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config7: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=model-inventory, waitUntilReady=true] 03:01:34,126 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config7: No shard found for model-inventory 03:01:34,127 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config7: Failed to find leader for shard model-inventory: received response: {} org.opendaylight.controller.cluster.datastore.exceptions.PrimaryNotFoundException: No primary shard found for model-inventory. at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.findPrimary(ShardManager.java:1056) at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.handleCommand(ShardManager.java:207) at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManagerTest$TestShardManager.handleCommand(ShardManagerTest.java:2198) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) at scala.PartialFunction.applyOrElse(PartialFunction.scala:214) at scala.PartialFunction.applyOrElse$(PartialFunction.scala:213) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:537) at akka.actor.Actor.aroundReceive$(Actor.scala:535) at akka.persistence.AbstractPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:295) at akka.persistence.Eventsourced$$anon$4.stateReceive(Eventsourced.scala:917) at akka.persistence.Eventsourced.aroundReceive(Eventsourced.scala:245) at akka.persistence.Eventsourced.aroundReceive$(Eventsourced.scala:244) at akka.persistence.AbstractPersistentActor.aroundReceive(PersistentActor.scala:295) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579) at akka.actor.ActorCell.invoke(ActorCell.scala:547) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) at akka.dispatch.Mailbox.run(Mailbox.scala:231) at akka.dispatch.Mailbox.exec(Mailbox.scala:243) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) 03:01:34,230 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$d#-330816222] 03:01:34,242 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config7 03:01:34,248 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingLocalReplicaLeader starting 03:01:34,254 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config8 03:01:34,255 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$e#-752579996] to Actor[akka://test/user/termination-monitor] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:34,257 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config8 03:01:34,257 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config8: Creating local shard: member-1-shard-default-config8 03:01:34,266 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:34,266 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config8 03:01:34,266 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config8: Initializing shard [default] 03:01:34,266 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-default-config8 is initialized 03:01:34,266 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config8: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config8, leaderId=member-1-shard-default-config8, leaderPayloadVersion=13] 03:01:34,267 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config8: Received role changed for member-1-shard-default-config8 from Candidate to Leader 03:01:34,267 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config8: All Shards are ready - data store config8 is ready 03:01:34,267 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config8: onAddShardReplica: AddShardReplica[ShardName=default] 03:01:34,270 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config8: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:34,270 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config8: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#-620451730, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$IHbtyKwr@34538dde] 03:01:34,271 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config8: Local shard default already exists 03:01:34,274 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config8: findLocalShard : default 03:01:34,274 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingLocalReplicaLeader ending 03:01:34,377 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$e#-752579996] 03:01:34,389 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Stopping ShardManager shard-manager-config8 03:01:34,394 PM [main] [INFO] ShardManagerTest - testOnReceiveFindLocalShardWaitForShardInitialized starting 03:01:34,401 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Starting ShardManager shard-manager-config9 03:01:34,402 PM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$f#-1964951164] to Actor[akka://test/user/termination-monitor] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:34,404 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config9 03:01:34,404 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config9: Creating local shard: member-1-shard-default-config9 03:01:34,412 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:34,413 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config9 03:01:34,413 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config9: findLocalShard : default 03:01:34,413 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config9: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config9, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 03:01:34,413 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config9: Initializing shard [default] 03:01:34,413 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config9 is initialized 03:01:34,413 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardInformation - Shard member-1-shard-default-config9 is initialized - notifying 1 OnShardInitialized callbacks 03:01:34,420 PM [main] [INFO] ShardManagerTest - testOnReceiveFindLocalShardWaitForShardInitialized starting 03:01:34,522 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$f#-1964951164] 03:01:34,534 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config9 03:01:34,540 PM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberUnreachableAndLeadershipChange starting 03:01:34,558 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 03:01:34,592 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [-1446201434749401983] 03:01:34,592 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 03:01:34,598 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 03:01:34,598 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 03:01:34,607 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:34,608 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:34,636 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 03:01:34,636 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config10] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:34,636 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 03:01:34,637 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 03:01:34,638 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config10 03:01:34,641 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Recovery complete : shard-manager-config10 03:01:34,641 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config10: Creating local shard: member-1-shard-default-config10 03:01:34,648 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 03:01:34,649 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 03:01:34,656 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 03:01:34,683 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-8730624853909137199] 03:01:34,687 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 03:01:34,689 PM [cluster-test-akka.actor.default-dispatcher-6] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:34,689 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 03:01:34,702 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:34,702 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:34,716 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Starting ShardManager shard-manager-config10 03:01:34,724 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:34,724 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config10 03:01:34,724 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 03:01:34,724 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 to akka://cluster-test/user/member-1-shard-default-config 03:01:34,724 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config10: updated peer member-1-shard-default-config10 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 on shard member-1-shard-default-config10 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#1044708275] 03:01:34,724 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config10: Initializing shard [default] 03:01:34,724 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-1-shard-default-config10 is initialized 03:01:34,724 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config10, leaderId=member-2-shard-default-config10, leaderPayloadVersion=13] 03:01:34,725 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: Received role changed for member-1-shard-default-config10 from Candidate to Follower 03:01:34,729 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config10] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:34,778 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Recovery complete : shard-manager-config10 03:01:34,779 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config10: Creating local shard: member-2-shard-default-config10 03:01:34,780 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:34,780 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config10 03:01:34,780 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config10: Initializing shard [default] 03:01:34,780 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-2-shard-default-config10 is initialized 03:01:34,780 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config10, leaderId=member-2-shard-default-config10, leaderPayloadVersion=13] 03:01:34,781 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: Received role changed for member-2-shard-default-config10 from Candidate to Leader 03:01:34,781 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 03:01:34,855 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 03:01:34,864 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 03:01:34,864 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 03:01:34,864 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 03:01:34,864 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config10 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config10/member-1-shard-default-config10 to akka://cluster-test/user/member-2-shard-default-config 03:01:34,865 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 03:01:35,613 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 03:01:35,614 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:35,614 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 03:01:35,614 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 to akka://cluster-test/user/member-1-shard-default-config 03:01:35,614 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 03:01:35,615 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:35,616 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shard-manager-config10: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10, primaryVersion=13] 03:01:35,619 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 03:01:35,619 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config10 as unavailable. 03:01:35,621 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:35,621 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config10: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config10, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config10, activeMember=true] 03:01:35,734 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config10: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:35,734 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 03:01:35,734 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config10 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config10/member-2-shard-default-config10 to akka://cluster-test/user/member-2-shard-default-config 03:01:35,734 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 03:01:36,840 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config10: Received ShardNotInitializedTimeout message for shard default 03:01:36,840 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config10: Returning NoShardLeaderException for shard default 03:01:36,841 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config10, leaderId=member-1-shard-default-config10, leaderPayloadVersion=13] 03:01:36,841 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 03:01:36,841 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: Received role changed for member-1-shard-default-config10 from Follower to Leader 03:01:36,842 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config10: All Shards are ready - data store config10 is ready 03:01:36,842 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config10: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:36,842 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config10: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/member-1-shard-default-config#1044708275, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$IHbtyKwr@7be03bb2] 03:01:36,847 PM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberUnreachableAndLeadershipChange ending 03:01:36,950 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - Stopping ShardManager shard-manager-config10 03:01:36,953 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:36,954 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:36,968 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:36,969 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:36,993 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shard-manager-config10 03:01:36,995 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:36,995 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:38,19 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/07/2024 15:01:38.023] [cluster-test-akka.actor.internal-dispatcher-2] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:38,45 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config11 03:01:38,47 PM [test-akka.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/shardmanager-config11] to Actor[akka://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:38,48 PM [test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shard-manager-config11 03:01:38,69 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:38,69 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config11: onAddShardReplica: AddShardReplica[ShardName=astronauts] 03:01:38,72 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config11: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-21#-1490458948 for shard member-1-shard-astronauts-config11 03:01:38,92 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config11: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-2}] 03:01:38,93 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config11: A shard replica operation for astronauts is already in progress 03:01:38,198 PM [main] [INFO] ShardManagerTest - testPerShardDatastoreContext starting 03:01:38,209 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config12 03:01:38,215 PM [test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from Actor[akka://test/user/$g#-799717575] to Actor[akka://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:38,217 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config12 03:01:38,217 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config12: Creating local shard: member-1-shard-default-config12 03:01:38,220 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config12: Creating local shard: member-1-shard-topology-config12 03:01:38,221 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:38,221 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config12 03:01:38,221 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-topology-config12 03:01:38,224 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-default-config12 03:01:38,225 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-topology-config12 03:01:38,275 PM [main] [INFO] ShardManagerTest - testPerShardDatastoreContext ending 03:01:38,377 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/default1#-637466647] 03:01:38,389 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/topology2#1587767207] 03:01:38,389 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$g#-799717575] 03:01:38,390 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config12 03:01:38,394 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForLocalLeaderShard starting 03:01:38,399 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config13 03:01:38,401 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config13 03:01:38,401 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config13: Creating local shard: member-1-shard-default-config13 03:01:38,410 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:38,411 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config13 03:01:38,411 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config13: Initializing shard [default] 03:01:38,411 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config13 is initialized 03:01:38,411 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config13: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config13, leaderId=member-1-shard-default-config13, leaderPayloadVersion=13] 03:01:38,461 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config13: Received role changed for member-1-shard-default-config13 from Candidate to Leader 03:01:38,461 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config13: All Shards are ready - data store config13 is ready 03:01:38,461 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config13: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 03:01:38,461 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config13: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#-552858221, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$IHbtyKwr@614c7175] 03:01:38,462 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForLocalLeaderShard ending 03:01:38,563 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$h#1028582605] 03:01:38,575 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config13 03:01:38,579 PM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberWithNameContainedInLeaderIdUnreachable starting 03:01:38,595 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 03:01:38,629 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2562] with UID [8117569664832906563] 03:01:38,629 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Starting up, Akka version [2.6.21] ... 03:01:38,633 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Registered cluster JMX MBean [akka:type=Cluster] 03:01:38,634 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Started up successfully 03:01:38,636 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:38,636 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:38,658 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Node [akka://cluster-test@127.0.0.1:2562] is JOINING itself (with roles [member-256, dc-default], version [0.0.0]) and forming new cluster 03:01:38,658 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - is the new leader among reachable nodes (more leaders may exist) 03:01:38,658 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Leader is moving node [akka://cluster-test@127.0.0.1:2562] to [Up] 03:01:38,660 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config14] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:38,662 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config14 03:01:38,673 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 03:01:38,678 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Recovery complete : shard-manager-config14 03:01:38,678 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config14: Creating local shard: member-256-shard-default-config14 03:01:38,679 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-256}, address: akka://cluster-test@127.0.0.1:2562 03:01:38,679 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 03:01:38,698 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [5700140966581355981] 03:01:38,720 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 03:01:38,720 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:38,720 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 03:01:38,720 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:38,720 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:38,725 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config14 03:01:38,725 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config14] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:38,727 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:38,727 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-256-shard-default-config14 03:01:38,727 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 03:01:38,727 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 to akka://cluster-test/user/member-256-shard-default-config 03:01:38,727 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config14: updated peer member-256-shard-default-config14 on member MemberName{name=member-256} with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 on shard member-256-shard-default-config14 whose actor address is Actor[akka://cluster-test/user/member-256-shard-default-config#-819855513] 03:01:38,728 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config14: Initializing shard [default] 03:01:38,728 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-256-shard-default-config14 is initialized 03:01:38,728 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-256-shard-default-config14, leaderId=member-256-shard-default-config14, leaderPayloadVersion=13] 03:01:38,728 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: Received role changed for member-256-shard-default-config14 from Candidate to Leader 03:01:38,728 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 03:01:38,761 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] ShardManager - Recovery complete : shard-manager-config14 03:01:38,761 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config14: Creating local shard: member-2-shard-default-config14 03:01:38,762 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:38,762 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config14 03:01:38,762 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardManager - shard-manager-config14: Initializing shard [default] 03:01:38,762 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardInformation - Shard member-2-shard-default-config14 is initialized 03:01:38,762 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config14: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config14, leaderId=member-256-shard-default-config14, leaderPayloadVersion=13] 03:01:38,762 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config14: Received role changed for member-2-shard-default-config14 from Candidate to Follower 03:01:38,808 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 03:01:38,813 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2562] 03:01:38,813 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-256}, address: akka://cluster-test@127.0.0.1:2562 03:01:38,814 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] ShardInformation - updatePeerAddress for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 03:01:38,814 PM [cluster-test-akka.actor.default-dispatcher-17] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-256-shard-default-config14 with address akka://cluster-test@127.0.0.1:2562/user/shardmanager-config14/member-256-shard-default-config14 to akka://cluster-test/user/member-2-shard-default-config 03:01:38,814 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 03:01:39,648 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2562] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 03:01:39,649 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:39,649 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 03:01:39,649 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 to akka://cluster-test/user/member-256-shard-default-config 03:01:39,649 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 03:01:39,649 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config14: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:39,650 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config14: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2562/user/member-256-shard-default-config#-819855513, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$IHbtyKwr@62b630e0] 03:01:39,651 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 03:01:39,651 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config14: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:39,651 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shard-manager-config14: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2562/user/member-256-shard-default-config#-819855513, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$IHbtyKwr@62b630e0] 03:01:39,653 PM [main] [INFO] ShardManagerTest - testShardAvailabilityChangeOnMemberWithNameContainedInLeaderIdUnreachable ending 03:01:39,740 PM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config14: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:39,740 PM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 03:01:39,740 PM [cluster-test-akka.actor.default-dispatcher-16] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config14 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config14/member-2-shard-default-config14 to akka://cluster-test/user/member-2-shard-default-config 03:01:39,740 PM [cluster-test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config14: All Shards are ready - data store config14 is ready 03:01:39,756 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shard-manager-config14 03:01:39,760 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:39,760 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:39,775 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:39,775 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:39,778 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 03:01:39,798 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shard-manager-config14 03:01:39,799 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:39,799 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:40,745 PM [cluster-test-akka.actor.default-dispatcher-16] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - is the new leader among reachable nodes (more leaders may exist) 03:01:40,829 PM [cluster-test-akka.actor.default-dispatcher-16] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/07/2024 15:01:40.832] [cluster-test-akka.actor.internal-dispatcher-3] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:40,856 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config15 03:01:40,858 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config15 03:01:40,858 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config15: Creating local shard: member-1-shard-default-config15 03:01:40,861 PM [main] [INFO] ShardManager - shard-manager-config15: Received role changed for member-1-shard-default-config15 from Follower to Leader 03:01:40,962 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$c] 03:01:40,974 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config15 03:01:40,983 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config16 03:01:40,985 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config16 03:01:40,985 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config16: Creating local shard: member-1-shard-default-config16 03:01:40,985 PM [main] [INFO] ShardManager - shard-manager-config16: Received role changed for member-1-shard-default-config16 from Candidate to Leader 03:01:40,986 PM [main] [INFO] ShardManager - shard-manager-config16: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config16, leaderId=member-1-shard-default-config16, leaderPayloadVersion=13] 03:01:40,986 PM [main] [INFO] ShardManager - shard-manager-config16: All Shards are ready - data store config16 is ready 03:01:41,88 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$d] 03:01:41,99 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config16 03:01:41,104 PM [main] [INFO] ShardManagerTest - testOnCreateShardWithLocalMemberNotInShardConfig starting 03:01:41,111 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config17 03:01:41,114 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config17 03:01:41,114 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:41,129 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config17: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@53c3d5a1] 03:01:41,132 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config17 doCreateShard: shardId: member-1-shard-foo-config17, memberNames: [MemberName{name=member-5}, MemberName{name=member-6}], peerAddresses: {}, isActiveMember: false 03:01:41,146 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config17: findLocalShard : foo 03:01:41,146 PM [test-shard-dispatcher-17] [INFO] Shard - Shard created : member-1-shard-foo-config17, persistent : true 03:01:41,146 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config17: Scheduling 60000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config17, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=false] 03:01:41,168 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$i/member-1-shard-foo-config17/member-1-shard-foo-config17-notifier#580945161 created and ready for shard:member-1-shard-foo-config17 03:01:41,170 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$i/member-1-shard-foo-config17/shard-member-1:snapshot-read#-1061423828] 03:01:41,183 PM [test-shard-dispatcher-17] [INFO] Shard - Starting recovery for member-1-shard-foo-config17 with journal batch size 1 03:01:41,198 PM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-foo-config17: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 03:01:41,201 PM [test-shard-dispatcher-17] [INFO] Shard - member-1-shard-foo-config17: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:01:41,201 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config17: Initializing shard [foo] 03:01:41,202 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-foo-config17 is initialized 03:01:41,202 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-foo-config17 is initialized - notifying 1 OnShardInitialized callbacks 03:01:41,203 PM [main] [INFO] ShardManagerTest - testOnCreateShardWithLocalMemberNotInShardConfig ending 03:01:41,211 PM [test-shard-dispatcher-17] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config17: onLeadershipChange, isLeader: false, hasLeader : false 03:01:41,212 PM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-foo-config17: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config17 is no longer the leader 03:01:41,212 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config17 , received role change from null to Follower 03:01:41,216 PM [test-shard-dispatcher-17] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 03:01:41,218 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config17 , registered listener akka://test/user/$i 03:01:41,218 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config17: Received role changed for member-1-shard-foo-config17 from null to Follower 03:01:41,306 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$i#1302187674] 03:01:41,308 PM [test-shard-dispatcher-17] [INFO] Shard - Stopping Shard member-1-shard-foo-config17 03:01:41,309 PM [test-shard-dispatcher-17] [DEBUG] MessageAssembler - member-1-shard-foo-config17 (Follower): Closing 03:01:41,310 PM [test-shard-dispatcher-17] [DEBUG] Shard - member-1-shard-foo-config17: Aborting 0 pending queued transactions 03:01:41,311 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config17 03:01:41,313 PM [main] [INFO] ShardManagerTest - testServerRemovedShardActorNotRunning starting 03:01:41,316 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config18 03:01:41,319 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config18 03:01:41,319 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config18: Creating local shard: member-1-shard-default-config18 03:01:41,319 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config18: Creating local shard: member-1-shard-people-config18 03:01:41,428 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config18: findLocalShard : people 03:01:41,428 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config18: Returning NotInitializedException for shard people 03:01:41,429 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config18: findLocalShard : default 03:01:41,429 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config18: Returning NotInitializedException for shard default 03:01:41,431 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config18 : Local Shard replica for shard default has been removed 03:01:41,431 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config18: persisting the shard list [people] 03:01:41,434 PM [main] [INFO] ShardManagerTest - testServerRemovedShardActorNotRunning ending 03:01:41,440 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config18 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 03:01:41,445 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config18: Successfully deleted prior snapshots 03:01:41,536 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$e] 03:01:41,548 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config18 03:01:41,556 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config19 03:01:41,558 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config19 03:01:41,558 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config19: Creating local shard: member-1-shard-default-config19 03:01:41,568 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:41,568 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config19 03:01:41,568 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config19: Initializing shard [default] 03:01:41,568 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config19 is initialized 03:01:41,568 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config19: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 03:01:41,568 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config19: Returning NoShardLeaderException for shard default 03:01:41,670 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$j#1862304380] 03:01:41,681 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config19 03:01:41,686 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForFollowerShardWithNoInitialLeaderId starting 03:01:41,690 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config20 03:01:41,692 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config20 03:01:41,692 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config20: Creating local shard: member-1-shard-default-config20 03:01:41,701 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:41,702 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config20 03:01:41,702 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config20: Initializing shard [default] 03:01:41,702 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-default-config20 is initialized 03:01:41,702 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config20: Received role changed for member-1-shard-default-config20 from Candidate to Follower 03:01:41,702 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config20: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 03:01:41,702 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config20: Returning NoShardLeaderException for shard default 03:01:41,703 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config20: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config20, leaderId=member-1-shard-default-config20, leaderPayloadVersion=13] 03:01:41,704 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config20: All Shards are ready - data store config20 is ready 03:01:41,704 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config20: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 03:01:41,704 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config20: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config#129348887, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$IHbtyKwr@31813ea8] 03:01:41,704 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForFollowerShardWithNoInitialLeaderId starting 03:01:41,805 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$k#-741581599] 03:01:41,817 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config20 03:01:41,826 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config21 03:01:41,827 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config21 03:01:41,847 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:41,847 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config21: onAddShardReplica: AddShardReplica[ShardName=astronauts] 03:01:41,848 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config21: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-39#1910789139 for shard member-1-shard-astronauts-config21 03:01:41,860 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config21: onAddShardReplica: AddShardReplica[ShardName=astronauts] 03:01:41,860 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config21: A shard replica operation for astronauts is already in progress 03:01:41,965 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithCandidateShard starting 03:01:41,970 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config22 03:01:41,972 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config22 03:01:41,972 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config22: Creating local shard: member-1-shard-default-config22 03:01:41,981 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:41,982 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config22 03:01:41,982 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: Initializing shard [default] 03:01:41,982 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config22 is initialized 03:01:41,982 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config22: Received role changed for member-1-shard-default-config22 from null to Candidate 03:01:41,982 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:41,982 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config22, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Candidate, leaderId=null, activeMember=true] 03:01:43,201 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: Received ShardNotInitializedTimeout message for shard default 03:01:43,201 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config22: Returning NoShardLeaderException for shard default 03:01:43,202 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithCandidateShard ending 03:01:43,304 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$l#1296522950] 03:01:43,315 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config22 03:01:43,320 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithAddServerReplyFailure starting 03:01:43,325 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config23 03:01:43,327 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config23 03:01:43,446 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:43,457 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config23: onAddShardReplica: AddShardReplica[ShardName=astronauts] 03:01:43,458 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config23: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-44#155134871 for shard member-1-shard-astronauts-config23 03:01:43,460 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config23: Received AddServerReply{status=TIMEOUT} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-44#155134871 03:01:43,460 PM [test-akka.actor.default-dispatcher-14] [WARN] ShardManager - shard-manager-config23: Leader failed to add shard replica astronauts with status TIMEOUT 03:01:43,460 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config23: findLocalShard : astronauts 03:01:43,460 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config23: Local shard astronauts not found - shards present: [] 03:01:43,461 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config23: onAddShardReplica: AddShardReplica[ShardName=astronauts] 03:01:43,462 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config23: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-44#155134871 for shard member-1-shard-astronauts-config23 03:01:43,463 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config23: Received AddServerReply{status=NO_LEADER} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-44#155134871 03:01:43,463 PM [test-akka.actor.default-dispatcher-14] [WARN] ShardManager - shard-manager-config23: Leader failed to add shard replica astronauts with status NO_LEADER 03:01:43,464 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithAddServerReplyFailure ending 03:01:43,565 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-astronauts-config#242262463] 03:01:43,576 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/shardmanager-config23] 03:01:43,577 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config23 03:01:43,584 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config24 03:01:43,586 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config24 03:01:43,586 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config24: Creating local shard: member-1-shard-default-config24 03:01:43,586 PM [main] [INFO] ShardManager - shard-manager-config24: Received role changed for member-1-shard-default-config24 from Candidate to Follower 03:01:43,586 PM [main] [INFO] ShardManager - shard-manager-config24 Received follower initial sync status for member-1-shard-default-config24 status sync done true 03:01:43,586 PM [main] [INFO] ShardManager - shard-manager-config24 Received follower initial sync status for member-1-shard-default-config24 status sync done false 03:01:43,688 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$f] 03:01:43,699 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config24 03:01:43,705 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config25 03:01:43,707 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config25 03:01:43,707 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config25: Creating local shard: member-1-shard-default-config25 03:01:43,708 PM [main] [INFO] ShardManager - shard-manager-config25: Received role changed for unknown from Candidate to Leader 03:01:43,810 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$g] 03:01:43,821 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config25 03:01:43,823 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithUninitializedShard starting 03:01:43,828 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config26 03:01:43,830 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config26 03:01:43,830 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config26: Creating local shard: member-1-shard-default-config26 03:01:43,839 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:43,839 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config26 03:01:43,839 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config26: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:43,839 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config26: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config26, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 03:01:44,450 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config26: Received ShardNotInitializedTimeout message for shard default 03:01:44,451 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config26: Returning NotInitializedException for shard default 03:01:44,452 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config26: Initializing shard [default] 03:01:44,452 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config26 is initialized 03:01:44,652 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithUninitializedShard ending 03:01:44,755 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$m#-1391881261] 03:01:44,767 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config26 03:01:44,778 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config27 03:01:44,780 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config27 03:01:44,780 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config27: Creating local shard: member-1-shard-default-config27 03:01:44,789 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:44,789 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config27 03:01:44,789 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config27: Initializing shard [default] 03:01:44,789 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-default-config27 is initialized 03:01:44,789 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config27: findLocalShard : default 03:01:44,892 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$n#-995219058] 03:01:44,903 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config27 03:01:44,906 PM [main] [INFO] ShardManagerTest - testRegisterForShardLeaderChanges starting 03:01:44,911 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config28 03:01:44,913 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config28 03:01:44,913 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config28: Creating local shard: member-1-shard-default-config28 03:01:44,922 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:44,922 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config28 03:01:44,922 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config28: Initializing shard [default] 03:01:44,922 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config28 is initialized 03:01:45,16 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config28: onRegisterForShardAvailabilityChanges: org.opendaylight.controller.cluster.datastore.shardmanager.RegisterForShardAvailabilityChanges@2cd8ca47 03:01:45,20 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=13] 03:01:45,31 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=13] 03:01:45,534 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=null, leaderPayloadVersion=13] 03:01:45,562 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-2-shard-default-config28, leaderPayloadVersion=13] 03:01:45,573 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config28: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config28, leaderId=member-1-shard-default-config28, leaderPayloadVersion=13] 03:01:46,73 PM [main] [INFO] ShardManagerTest - testRegisterForShardLeaderChanges ending 03:01:46,176 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$o#-1870040092] 03:01:46,187 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config28 03:01:46,195 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config29 03:01:46,198 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config29 03:01:46,198 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config29: Creating local shard: member-1-shard-default-config29 03:01:46,301 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$h] 03:01:46,312 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config29 03:01:46,320 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config30 03:01:46,322 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config30 03:01:46,322 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config30: Creating local shard: member-1-shard-default-config30 03:01:46,332 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:46,332 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config30 03:01:46,332 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config30: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=non-existent, waitUntilReady=false] 03:01:46,332 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config30: No shard found for non-existent 03:01:46,434 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$p#-716913988] 03:01:46,446 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config30 03:01:46,455 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config31 03:01:46,458 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config31 03:01:46,466 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config31: onAddShardReplica: AddShardReplica[ShardName=model-inventory] 03:01:46,466 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config31: No module configuration exists for shard model-inventory 03:01:46,568 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$q#-140962594] 03:01:46,579 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config31 03:01:46,582 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingReplicaInRemoteShardLeader starting 03:01:46,587 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config32 03:01:46,590 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config32 03:01:46,590 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Creating local shard: member-1-shard-default-config32 03:01:46,598 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:46,598 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config32 03:01:46,598 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: Initializing shard [default] 03:01:46,598 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config32 is initialized 03:01:46,599 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config32: Received MemberUp: memberName: MemberName{name=leader-member}, address: akka://test 03:01:46,599 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer leader-member-shard-default-config32 with address akka://test/user/shardmanager-config32/leader-member-shard-default-config32 03:01:46,599 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer leader-member-shard-default-config32 with address akka://test/user/shardmanager-config32/leader-member-shard-default-config32 to akka://test/user/member-1-shard-default-config 03:01:46,599 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config32: Received role changed for member-1-shard-default-config32 from Candidate to Follower 03:01:46,599 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config32: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config32, leaderId=leader-member-shard-default-config32, leaderPayloadVersion=13] 03:01:46,600 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config32: All Shards are ready - data store config32 is ready 03:01:46,600 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 03:01:46,600 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:46,600 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=13] 03:01:46,601 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 03:01:46,602 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Received AddServerReply{status=ALREADY_EXISTS} for shard default from leader akka://test/user/shardmanager-config32/leader-member-shard-default-config32 03:01:46,602 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Local shard default already exists 03:01:46,650 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: findLocalShard : default 03:01:46,650 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 03:01:46,651 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:46,651 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=13] 03:01:46,652 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 03:01:46,653 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config32: Received AddServerReply{status=ALREADY_EXISTS} for shard default from leader akka://test/user/shardmanager-config32/leader-member-shard-default-config32 03:01:46,653 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config32: Local shard default already exists 03:01:46,656 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-default-config32 03:01:46,656 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: onAddShardReplica: AddShardReplica[ShardName=default] 03:01:46,657 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:46,657 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config32/leader-member-shard-default-config32, primaryVersion=13] 03:01:46,657 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config32: Sending AddServer message to peer akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for shard member-1-shard-default-config32 03:01:46,771 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config32: AddServer request to akka://test/user/shardmanager-config32/leader-member-shard-default-config32 for default failed akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://test/), Path(/user/shardmanager-config32/leader-member-shard-default-config32)]] after [100 ms]. Message of type [org.opendaylight.controller.cluster.raft.messages.AddServer]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. 03:01:46,772 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config32: findLocalShard : default 03:01:46,772 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithPreExistingReplicaInRemoteShardLeader ending 03:01:46,874 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/shardmanager-config32] 03:01:46,886 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config32 03:01:46,889 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForRemoteShard starting 03:01:46,904 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 03:01:46,918 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [-660335755640659281] 03:01:46,918 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 03:01:46,924 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 03:01:46,924 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 03:01:46,926 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:46,926 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:46,957 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 03:01:46,957 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 03:01:46,957 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 03:01:46,963 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config33 03:01:46,964 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config33] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:46,983 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 03:01:46,990 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config33 03:01:46,990 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-1-shard-default-config33 03:01:46,992 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 03:01:46,992 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 03:01:47,17 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [6393250911009451437] 03:01:47,17 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 03:01:47,17 PM [cluster-test-akka.actor.default-dispatcher-5] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:47,17 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 03:01:47,30 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:47,30 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:47,42 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:47,42 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config33 03:01:47,42 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 03:01:47,43 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 to akka://test/user/member-1-shard-default-config 03:01:47,43 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: updated peer member-1-shard-default-config33 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 on shard member-1-shard-default-config33 whose actor address is Actor[akka://test/user/member-1-shard-default-config#-1573070053] 03:01:47,47 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config33] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:47,60 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config33 03:01:47,73 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Recovery complete : shard-manager-config33 03:01:47,74 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-2-shard-default-config33 03:01:47,74 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config33: Creating local shard: member-2-shard-astronauts-config33 03:01:47,75 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:47,75 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config33 03:01:47,75 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - Creating Shard member-2-shard-astronauts-config33 03:01:47,75 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardManager - shard-manager-config33: Initializing shard [astronauts] 03:01:47,75 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Shard member-2-shard-astronauts-config33 is initialized 03:01:47,75 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config33: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-astronauts-config33, leaderId=member-2-shard-astronauts-config33, leaderPayloadVersion=12] 03:01:47,75 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shard-manager-config33: Received role changed for member-2-shard-astronauts-config33 from Candidate to Leader 03:01:47,126 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 03:01:47,130 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 03:01:47,131 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 03:01:47,131 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 03:01:47,131 PM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-default-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 03:01:47,131 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-astronauts-config33 03:01:47,131 PM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33/member-1-shard-astronauts-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 03:01:47,939 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 03:01:47,940 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:47,940 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 03:01:47,940 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 to akka://test/user/member-1-shard-default-config 03:01:47,941 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=false] 03:01:47,941 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: findPrimary for astronauts forwarding to remote ShardManager akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33, visitedAddresses: [akka://cluster-test@127.0.0.1:2558/user/shardmanager-config33] 03:01:47,954 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.RemoteFindPrimary [shardName=astronauts, waitUntilReady=false] 03:01:47,954 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config33: Found primary for astronauts: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config#625796626, primaryVersion=12] 03:01:47,959 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForRemoteShard ending 03:01:48,51 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config33: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:48,51 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 03:01:48,51 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-default-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 03:01:48,51 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-astronauts-config33 03:01:48,51 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config33 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config33/member-2-shard-astronauts-config33 to akka://cluster-test/user/member-2-shard-astronauts-config 03:01:48,61 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config33 03:01:48,62 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:48,62 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:48,73 PM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:48,73 PM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:48,100 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config33 03:01:48,102 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:48,102 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:49,123 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/07/2024 15:01:49.128] [cluster-test-akka.actor.internal-dispatcher-4] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:49,144 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config34 03:01:49,146 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config34 03:01:49,146 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config34: Creating local shard: member-1-shard-default-config34 03:01:49,157 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config34: findLocalShard : default 03:01:49,157 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config34: Returning NotInitializedException for shard default 03:01:49,259 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$r#1786064943] 03:01:49,270 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config34 03:01:49,279 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config35 03:01:49,281 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config35 03:01:49,281 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config35: Creating local shard: member-1-shard-default-config35 03:01:49,287 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:49,287 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config35 03:01:49,287 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config35: Initializing shard [default] 03:01:49,287 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config35 is initialized 03:01:49,287 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config35: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config35, leaderId=member-1-shard-default-config35, leaderPayloadVersion=13] 03:01:49,288 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config35: Received role changed for member-1-shard-default-config35 from Candidate to Leader 03:01:49,288 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config35: All Shards are ready - data store config35 is ready 03:01:49,288 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config35: onChangeShardServersVotingStatus: ChangeShardMembersVotingStatus [shardName=default, meberVotingStatusMap={member-2=true}] 03:01:49,288 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config35: findLocalShard : default 03:01:49,288 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config35: Sending ChangeServersVotingStatus message ChangeServersVotingStatus [serverVotingStatusMap={member-2-shard-default-config35=true}, serversVisited=[]] to local shard akka://test/user/member-1-shard-default-config35 03:01:49,289 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config35: Received ServerChangeReply{status=OK} from local shard akka://test/user/member-1-shard-default-config35 03:01:49,289 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config35: ChangeServersVotingStatus succeeded for shard default 03:01:49,440 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config35#259650086] 03:01:49,454 PM [main] [INFO] ShardManagerTest - testShutDown starting 03:01:49,460 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config36 03:01:49,462 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config36 03:01:49,462 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: Creating local shard: member-1-shard-shard2-config36 03:01:49,463 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: Creating local shard: member-1-shard-shard1-config36 03:01:49,472 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:49,472 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config36 03:01:49,472 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config36 03:01:49,472 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config36: Initializing shard [shard1] 03:01:49,472 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config36 is initialized 03:01:49,472 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config36: Initializing shard [shard2] 03:01:49,472 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config36 is initialized 03:01:49,472 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config36: Issuing gracefulStop to shard member-1-shard-shard2-config36 03:01:49,473 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config36: Issuing gracefulStop to shard member-1-shard-shard1-config36 03:01:49,473 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Shutting down ShardManager shard-manager-config36 - waiting on 2 shards 03:01:50,25 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-shard1-config36#814432834] 03:01:50,26 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-shard2-config36#1286429443] 03:01:50,26 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config36: All shards shutdown - sending PoisonPill to self 03:01:50,27 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config36 03:01:50,27 PM [main] [INFO] ShardManagerTest - testShutDown ending 03:01:50,129 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$t#-1264426088] 03:01:50,142 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShardBeforeMemberUp starting 03:01:50,146 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config37 03:01:50,147 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config37 03:01:50,147 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config37: Creating local shard: member-1-shard-default-config37 03:01:50,157 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:50,157 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config37 03:01:50,157 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config37: Initializing shard [default] 03:01:50,157 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config37 is initialized 03:01:50,157 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config37: Received role changed for member-1-shard-default-config37 from Candidate to Follower 03:01:50,157 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerTest$TestShardManager - Received unhandled message LeaderStateChanged [memberId=member-1-shard-default-config37, leaderId=member-2-shard-default-config37, leaderPayloadVersion=13] 03:01:50,158 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config37: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 03:01:50,158 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config37: Returning NoShardLeaderException for shard default 03:01:50,159 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShardBeforeMemberUp ending 03:01:50,260 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$u#2118814783] 03:01:50,272 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config37 03:01:50,275 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithNoRoleShard starting 03:01:50,279 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config38 03:01:50,280 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config38 03:01:50,280 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config38: Creating local shard: member-1-shard-default-config38 03:01:50,289 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:50,289 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config38 03:01:50,289 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config38: Initializing shard [default] 03:01:50,289 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config38 is initialized 03:01:50,290 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config38: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:50,290 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config38: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config38, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 03:01:51,500 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config38: Received ShardNotInitializedTimeout message for shard default 03:01:51,500 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config38: Returning NoShardLeaderException for shard default 03:01:51,502 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithNoRoleShard ending 03:01:51,603 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$v#313063387] 03:01:51,605 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config38 03:01:51,608 PM [main] [INFO] ShardManagerTest - testShardPersistenceWithRestoredData starting 03:01:51,613 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config39 03:01:51,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config39: onSnapshotOffer: ShardManagerSnapshot [ShardList = [default, astronauts] ] 03:01:51,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config39: adding shard astronauts 03:01:51,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config39: removing shard people 03:01:51,615 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config39 03:01:51,616 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Creating local shard: member-1-shard-default-config39 03:01:51,616 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Creating local shard: member-1-shard-astronauts-config39 03:01:51,628 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : people 03:01:51,628 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Local shard people not found - shards present: [default, astronauts] 03:01:51,628 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : default 03:01:51,628 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Returning NotInitializedException for shard default 03:01:51,628 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: findLocalShard : astronauts 03:01:51,628 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config39: Returning NotInitializedException for shard astronauts 03:01:51,628 PM [main] [INFO] ShardManagerTest - testShardPersistenceWithRestoredData ending 03:01:51,730 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$i] 03:01:51,741 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config39 03:01:51,745 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShard starting 03:01:51,749 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shard-manager-config40 03:01:51,751 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config40 03:01:51,751 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config40: Creating local shard: member-1-shard-default-config40 03:01:51,760 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:51,760 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config40 03:01:51,760 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config40: Initializing shard [default] 03:01:51,761 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config40 is initialized 03:01:51,761 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config40: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 03:01:51,761 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config40 with address akka://test/user/shardmanager-config40/member-2-shard-default-config40 03:01:51,761 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config40 with address akka://test/user/shardmanager-config40/member-2-shard-default-config40 to akka://test/user/member-1-shard-default-config 03:01:51,761 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config40: Received role changed for member-1-shard-default-config40 from Candidate to Follower 03:01:51,761 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config40: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config40, leaderId=member-2-shard-default-config40, leaderPayloadVersion=12] 03:01:51,761 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config40: All Shards are ready - data store config40 is ready 03:01:51,761 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config40: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 03:01:51,761 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config40: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://test/user/shardmanager-config40/member-2-shard-default-config40, primaryVersion=12] 03:01:51,762 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryForNonLocalLeaderShard ending 03:01:51,864 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$w#427418386] 03:01:51,875 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config40 03:01:51,878 PM [main] [INFO] ShardManagerTest - testShardAvailabilityOnChangeOfMemberReachability starting 03:01:51,895 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 03:01:51,922 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [2161566777039239531] 03:01:51,922 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 03:01:51,930 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 03:01:51,930 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 03:01:51,930 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:51,930 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:51,948 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 03:01:51,953 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config41] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:51,962 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 03:01:51,962 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 03:01:51,972 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 03:01:51,976 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config41 03:01:51,982 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config41 03:01:51,982 PM [cluster-test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config41: Creating local shard: member-1-shard-default-config41 03:01:51,983 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 03:01:51,983 PM [cluster-test-akka.actor.default-dispatcher-15] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 03:01:51,996 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-807693803764826256] 03:01:51,996 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 03:01:52,1 PM [cluster-test-akka.actor.default-dispatcher-10] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:52,1 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 03:01:52,7 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:52,7 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:52,19 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ShardManager - Starting ShardManager shard-manager-config41 03:01:52,20 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config41] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:52,34 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:52,34 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config41 03:01:52,34 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 03:01:52,34 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 03:01:52,34 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config41: updated peer member-1-shard-default-config41 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 on shard member-1-shard-default-config41 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#-2077742674] 03:01:52,34 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config41: Initializing shard [default] 03:01:52,34 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is initialized 03:01:52,34 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config41: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config41, leaderId=member-2-shard-default-config41, leaderPayloadVersion=13] 03:01:52,34 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config41: Received role changed for member-1-shard-default-config41 from Candidate to Follower 03:01:52,39 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config41 03:01:52,39 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Creating local shard: member-2-shard-default-config41 03:01:52,79 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:52,79 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config41 03:01:52,79 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config41: Initializing shard [default] 03:01:52,79 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-2-shard-default-config41 is initialized 03:01:52,79 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-default-config41, leaderId=member-2-shard-default-config41, leaderPayloadVersion=13] 03:01:52,79 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received role changed for member-2-shard-default-config41 from Candidate to Leader 03:01:52,79 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 03:01:52,120 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 03:01:52,121 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 03:01:52,121 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 03:01:52,121 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-1-shard-default-config41 to akka://cluster-test/user/member-2-shard-default-config 03:01:52,121 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 03:01:52,121 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 03:01:52,946 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 03:01:52,950 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:52,950 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 03:01:52,950 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 03:01:52,950 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 03:01:52,951 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:52,951 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=13] 03:01:52,952 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 03:01:52,952 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as unavailable. 03:01:52,952 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shard-manager-config41: Received MemberRemoved: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 03:01:52,952 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:52,952 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config41: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config41, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config41, activeMember=true] 03:01:53,31 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:53,31 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 03:01:53,31 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-2-shard-default-config 03:01:53,31 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 03:01:54,165 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config41: Received ShardNotInitializedTimeout message for shard default 03:01:54,165 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config41: Returning NoShardLeaderException for shard default 03:01:54,165 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Received ReachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 03:01:54,165 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 03:01:54,165 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 03:01:54,166 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as available. 03:01:54,166 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:54,166 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=13] 03:01:54,166 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config41: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 03:01:54,166 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 03:01:54,166 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 03:01:54,166 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - shard-manager-config41: All Shards are ready - data store config41 is ready 03:01:54,166 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Received UnreachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 03:01:54,166 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as unavailable. 03:01:54,167 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config41: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:54,167 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config41: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config41, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=Follower, leaderId=member-2-shard-default-config41, activeMember=true] 03:01:54,167 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Received ReachableMember: memberName MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2558 03:01:54,167 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 03:01:54,167 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config41 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41 to akka://cluster-test/user/member-1-shard-default-config 03:01:54,167 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is initialized - notifying 1 OnShardInitialized callbacks 03:01:54,167 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Marking Leader member-2-shard-default-config41 as available. 03:01:54,167 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardInformation - Shard member-1-shard-default-config41 is ready - notifying 1 OnShardInitialized callbacks 03:01:54,167 PM [cluster-test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config41: Found primary for default: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2558/user/shardmanager-config41/member-2-shard-default-config41, primaryVersion=13] 03:01:54,167 PM [main] [INFO] ShardManagerTest - testShardAvailabilityOnChangeOfMemberReachability ending 03:01:54,269 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config41 03:01:54,271 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:54,271 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:54,281 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:54,284 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:54,285 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 03:01:54,308 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config41 03:01:54,311 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:54,311 PM [cluster-test-akka.actor.default-dispatcher-17] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:55,322 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/07/2024 15:01:55.327] [cluster-test-akka.actor.internal-dispatcher-3] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:55,332 PM [main] [INFO] ShardManagerTest - testAddShardReplica starting 03:01:55,344 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started 03:01:55,359 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2558] with UID [1156512043796867753] 03:01:55,359 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Starting up, Akka version [2.6.21] ... 03:01:55,363 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Registered cluster JMX MBean [akka:type=Cluster] 03:01:55,363 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Started up successfully 03:01:55,370 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:55,370 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:55,380 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2558] is JOINING itself (with roles [member-1, dc-default], version [0.0.0]) and forming new cluster 03:01:55,380 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - is the new leader among reachable nodes (more leaders may exist) 03:01:55,381 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2558] to [Up] 03:01:55,382 PM [cluster-test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config42] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:55,384 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shardmanager-config42 03:01:55,396 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - Recovery complete : shardmanager-config42 03:01:55,396 PM [cluster-test-akka.actor.default-dispatcher-11] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-1-shard-default-config42 03:01:55,397 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 03:01:55,397 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 03:01:55,411 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started 03:01:55,433 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cluster-test@127.0.0.1:2559] with UID [-6631774365356233168] 03:01:55,434 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Starting up, Akka version [2.6.21] ... 03:01:55,444 PM [cluster-test-akka.actor.default-dispatcher-6] [WARN] Cluster - Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:55,444 PM [cluster-test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Started up successfully 03:01:55,448 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:01:55,448 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:01:55,467 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Starting ShardManager shardmanager-config42 03:01:55,470 PM [cluster-test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://cluster-test/user/shardmanager-config42] to Actor[akka://cluster-test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cluster-test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:01:55,490 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:55,490 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config42 03:01:55,490 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 03:01:55,490 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 to akka://cluster-test/user/member-1-shard-default-config 03:01:55,490 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: updated peer member-1-shard-default-config42 on member MemberName{name=member-1} with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 on shard member-1-shard-default-config42 whose actor address is Actor[akka://cluster-test/user/member-1-shard-default-config#1852200534] 03:01:55,500 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shardmanager-config42 03:01:55,500 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-2-shard-default-config42 03:01:55,501 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: Creating local shard: member-2-shard-astronauts-config42 03:01:55,501 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:55,502 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-2-shard-default-config42 03:01:55,502 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-2-shard-astronauts-config42 03:01:55,502 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shardmanager-config42: Initializing shard [astronauts] 03:01:55,502 PM [cluster-test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-2-shard-astronauts-config42 is initialized 03:01:55,502 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shardmanager-config42: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-2-shard-astronauts-config42, leaderId=member-2-shard-astronauts-config42, leaderPayloadVersion=12] 03:01:55,502 PM [cluster-test-akka.actor.default-dispatcher-14] [INFO] ShardManager - shardmanager-config42: Received role changed for member-2-shard-astronauts-config42 from Candidate to Leader 03:01:55,560 PM [cluster-test-akka.actor.default-dispatcher-13] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Node [akka://cluster-test@127.0.0.1:2559] is JOINING, roles [member-2, dc-default], version [0.0.0] 03:01:55,565 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2559] - Welcome from [akka://cluster-test@127.0.0.1:2558] 03:01:55,566 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-1}, address: akka://cluster-test@127.0.0.1:2558 03:01:55,566 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 03:01:55,566 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-default-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-default-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 03:01:55,566 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardInformation - updatePeerAddress for peer member-1-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-astronauts-config42 03:01:55,566 PM [cluster-test-akka.actor.default-dispatcher-12] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-1-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42/member-1-shard-astronauts-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 03:01:56,383 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://cluster-test@127.0.0.1:2558] - Leader is moving node [akka://cluster-test@127.0.0.1:2559] to [Up] 03:01:56,384 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:56,384 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 03:01:56,384 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 to akka://cluster-test/user/member-1-shard-default-config 03:01:56,387 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42: onAddShardReplica: AddShardReplica[ShardName=astronauts] 03:01:56,388 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=true] 03:01:56,388 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42: findPrimary for astronauts forwarding to remote ShardManager akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42, visitedAddresses: [akka://cluster-test@127.0.0.1:2558/user/shardmanager-config42] 03:01:56,392 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.RemoteFindPrimary [shardName=astronauts, waitUntilReady=true] 03:01:56,392 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: Found primary for astronauts: RemotePrimaryShardFound [primaryPath=akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config, primaryVersion=12] 03:01:56,395 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42: Sending AddServer message to peer akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config for shard member-1-shard-astronauts-config42 03:01:56,408 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: Received AddServerReply{status=OK, leaderHint=member-2-shard-astronauts-config42} for shard astronauts from leader akka://cluster-test@127.0.0.1:2559/user/member-2-shard-astronauts-config 03:01:56,408 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: Leader shard successfully added the replica shard astronauts 03:01:56,409 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-astronauts-config42 03:01:56,409 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shardmanager-config42: persisting the shard list [default, astronauts] 03:01:56,410 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 03:01:56,410 PM [cluster-test-akka.actor.default-dispatcher-13] [DEBUG] ShardManager - shardmanager-config42: Successfully deleted prior snapshots 03:01:56,438 PM [main] [INFO] ShardManagerTest - testAddShardReplica ending 03:01:56,465 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shardmanager-config42: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://cluster-test@127.0.0.1:2559 03:01:56,465 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 03:01:56,465 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-default-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-default-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 03:01:56,465 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-astronauts-config42 03:01:56,465 PM [cluster-test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Sending PeerAddressResolved for peer member-2-shard-astronauts-config42 with address akka://cluster-test@127.0.0.1:2559/user/shardmanager-config42/member-2-shard-astronauts-config42 to akka://cluster-test/user/member-2-shard-astronauts-config 03:01:56,540 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] ShardManager - Stopping ShardManager shardmanager-config42 03:01:56,542 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:56,542 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:56,558 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], control stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:56,558 PM [cluster-test-akka.actor.default-dispatcher-12] [WARN] Materializer - [outbound connection to [akka://cluster-test@127.0.0.1:2559], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted 03:01:56,561 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. 03:01:56,577 PM [cluster-test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shardmanager-config42 03:01:56,578 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:01:56,578 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:01:57,600 PM [cluster-test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/07/2024 15:01:57.603] [cluster-test-akka.actor.internal-dispatcher-4] [Cluster(akka://cluster-test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config 03:01:57,633 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config43 03:01:57,637 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config43 03:01:57,637 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config43: Creating local shard: member-1-shard-default-config43 03:01:57,642 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:57,642 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config43 03:01:57,642 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config43: Initializing shard [default] 03:01:57,642 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-default-config43 is initialized 03:01:57,642 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config43: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config43, leaderId=member-1-shard-default-config43, leaderPayloadVersion=13] 03:01:57,643 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config43: Received role changed for member-1-shard-default-config43 from Candidate to Leader 03:01:57,643 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config43: All Shards are ready - data store config43 is ready 03:01:57,643 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config43: onRemoveShardReplica: RemoveShardReplica [shardName=default, memberName=MemberName{name=member-1}] 03:01:57,643 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config43: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:57,643 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config43: Found primary for default: LocalPrimaryShardFound [primaryPath=akka://test@127.0.0.1:2565/user/member-1-shard-default-config43#-326681524, localShardDataTree=org.opendaylight.yangtools.yang.data.tree.api.DataTree$MockitoMock$IHbtyKwr@11ae16d8] 03:01:57,643 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Sending RemoveServer message to peer akka://test@127.0.0.1:2565/user/member-1-shard-default-config43#-326681524 for shard member-1-shard-default-config43 03:01:57,644 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Received RemoveServerReply{status=OK} for shard default 03:01:57,644 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config43: Leader shard successfully removed the replica shard default 03:01:57,795 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config43#-326681524] 03:01:57,809 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithIsolatedLeaderShard starting 03:01:57,813 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config44 03:01:57,815 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config44 03:01:57,815 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config44: Creating local shard: member-1-shard-default-config44 03:01:57,824 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:57,824 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config44 03:01:57,824 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config44: Initializing shard [default] 03:01:57,824 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-default-config44 is initialized 03:01:57,824 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config44: Received role changed for member-1-shard-default-config44 from null to IsolatedLeader 03:01:57,824 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config44: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=true] 03:01:57,824 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config44: Scheduling 1200 ms timer to wait for shard ShardInformation [shardId=member-1-shard-default-config44, leaderAvailable=false, actorInitialized=true, followerSyncStatus=false, role=IsolatedLeader, leaderId=null, activeMember=true] 03:01:59,41 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config44: Received ShardNotInitializedTimeout message for shard default 03:01:59,41 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config44: Returning NoShardLeaderException for shard default 03:01:59,42 PM [main] [INFO] ShardManagerTest - testOnReceiveFindPrimaryWaitForReadyWithIsolatedLeaderShard ending 03:01:59,143 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$y#-1555118418] 03:01:59,155 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config44 03:01:59,161 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config45 03:01:59,163 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Recovery complete : shard-manager-config45 03:01:59,163 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config45: Creating local shard: member-1-shard-default-config45 03:01:59,172 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:59,172 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config45 03:01:59,172 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config45: findLocalShard : non-existent 03:01:59,172 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config45: Local shard non-existent not found - shards present: [default] 03:01:59,274 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$z#810351591] 03:01:59,286 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config45 03:01:59,292 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config46 03:01:59,294 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config46 03:01:59,314 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:59,314 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config46: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-2}] 03:01:59,315 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config46: Sending RemoveServer message to peer akka://test@127.0.0.1:2565/system/testProbe-86#-557365422 for shard member-2-shard-astronauts-config46 03:01:59,326 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config46: onRemoveShardReplica: RemoveShardReplica [shardName=astronauts, memberName=MemberName{name=member-3}] 03:01:59,327 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config46: A shard replica operation for astronauts is already in progress 03:01:59,435 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config47 03:01:59,436 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config47 03:01:59,436 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config47: Creating local shard: member-1-shard-default-config47 03:01:59,445 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config47: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=default, waitUntilReady=false] 03:01:59,445 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config47: Returning NotInitializedException for shard default 03:01:59,547 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$A#449461808] 03:01:59,558 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Stopping ShardManager shard-manager-config47 03:01:59,561 PM [main] [INFO] ShardManagerTest - testServerRemovedShardActorRunning starting 03:01:59,568 PM [test-akka.actor.default-dispatcher-14] [INFO] ShardManager - Starting ShardManager shard-manager-config48 03:01:59,569 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Recovery complete : shard-manager-config48 03:01:59,569 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48: Creating local shard: member-1-shard-default-config48 03:01:59,569 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48: Creating local shard: member-1-shard-people-config48 03:01:59,678 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:59,678 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-default-config48 03:01:59,678 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - Creating Shard member-1-shard-people-config48 03:01:59,680 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48: Initializing shard [default] 03:01:59,680 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-default-config48 is initialized 03:01:59,680 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48: findLocalShard : people 03:01:59,680 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-people-config48, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 03:01:59,684 PM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-people-config48, persistent : true 03:01:59,689 PM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-people-config48 with journal batch size 1 03:01:59,689 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$j/member-1-shard-people-config48/shard-member-1:snapshot-read#-1344375406] 03:01:59,690 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$j/member-1-shard-people-config48/member-1-shard-people-config48-notifier#738742741 created and ready for shard:member-1-shard-people-config48 03:01:59,691 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-people-config48: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 03:01:59,691 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-people-config48: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:01:59,692 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-people-config48: onLeadershipChange, isLeader: false, hasLeader : false 03:01:59,692 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-people-config48: onStateChanged: Closing all transaction chains because shard member-1-shard-people-config48 is no longer the leader 03:01:59,692 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48: Initializing shard [people] 03:01:59,692 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-people-config48 is initialized 03:01:59,692 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-people-config48 is initialized - notifying 1 OnShardInitialized callbacks 03:01:59,692 PM [test-akka.actor.default-dispatcher-15] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-people-config48 , received role change from null to Follower 03:01:59,692 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config48: findLocalShard : default 03:01:59,692 PM [test-akka.actor.default-dispatcher-15] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-people-config48 , registered listener akka://test/user/$$j 03:01:59,692 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config48: Received role changed for member-1-shard-people-config48 from null to Follower 03:01:59,693 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48 : Sending Shutdown to Shard actor Actor[akka://test/user/member-1-shard-default-config48#160316197] with 10000 ms timeout 03:01:59,696 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48 : Local Shard replica for shard default has been removed 03:01:59,696 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48: persisting the shard list [people] 03:01:59,696 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 03:01:59,696 PM [main] [INFO] ShardManagerTest - testServerRemovedShardActorRunning ending 03:01:59,697 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config48: Successfully deleted prior snapshots 03:01:59,798 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/member-1-shard-default-config48#160316197] 03:01:59,809 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config48 : Successfully shut down Shard actor Actor[akka://test/user/member-1-shard-default-config48#160316197] 03:01:59,810 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$j] 03:01:59,811 PM [test-shard-dispatcher-18] [INFO] Shard - Stopping Shard member-1-shard-people-config48 03:01:59,811 PM [test-shard-dispatcher-18] [DEBUG] MessageAssembler - member-1-shard-people-config48 (Follower): Closing 03:01:59,812 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-people-config48: Aborting 0 pending queued transactions 03:01:59,812 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Stopping ShardManager shard-manager-config48 03:01:59,818 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config49 03:01:59,820 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config49 03:01:59,820 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config49: Creating local shard: member-1-shard-default-config49 03:01:59,821 PM [main] [INFO] ShardManager - shard-manager-config49: Received role changed for member-1-shard-default-config49 from null to Follower 03:01:59,821 PM [main] [INFO] ShardManager - shard-manager-config49: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-default-config49, leaderId=member-2-shard-default-config49, leaderPayloadVersion=13] 03:01:59,832 PM [main] [INFO] ShardManager - shard-manager-config49: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://test 03:01:59,832 PM [main] [INFO] ShardInformation - updatePeerAddress for peer member-2-shard-default-config49 with address akka://test/user/shardmanager-config49/member-2-shard-default-config49 03:01:59,832 PM [main] [INFO] ShardManager - shard-manager-config49: All Shards are ready - data store config49 is ready 03:01:59,933 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$k] 03:01:59,944 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config49 03:01:59,948 PM [main] [INFO] ShardManagerTest - testRestoreFromSnapshot starting 03:01:59,951 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Starting ShardManager shard-manager-config50 03:01:59,953 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config50 03:01:59,953 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: Restoring from ShardManagerSnapshot: ShardManagerSnapshot [ShardList = [shard1, shard2, astronauts] ] 03:01:59,953 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: onSnapshotOffer: ShardManagerSnapshot [ShardList = [shard1, shard2, astronauts] ] 03:01:59,953 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: adding shard shard1 03:01:59,953 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: adding shard shard2 03:01:59,953 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: adding shard astronauts 03:01:59,953 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-shard2-config50 03:01:59,954 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-astronauts-config50 03:01:59,954 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: Creating local shard: member-1-shard-shard1-config50 03:01:59,955 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:01:59,955 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config50 03:01:59,956 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-astronauts-config50 03:01:59,957 PM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-shard2-config50, persistent : true 03:01:59,958 PM [test-akka.actor.default-dispatcher-15] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-shard2-config50/member-1-shard-shard2-config50-notifier#402646728 created and ready for shard:member-1-shard-shard2-config50 03:01:59,958 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-shard2-config50/shard-member-1:snapshot-read#-65407602] 03:01:59,959 PM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-shard2-config50 with journal batch size 1 03:01:59,961 PM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-astronauts-config50, persistent : true 03:01:59,962 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-astronauts-config50/member-1-shard-astronauts-config50-notifier#-713146723 created and ready for shard:member-1-shard-astronauts-config50 03:01:59,962 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-astronauts-config50/shard-member-1:snapshot-read#1523130234] 03:01:59,962 PM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-astronauts-config50 with journal batch size 1 03:01:59,963 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config50 03:01:59,964 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 03:01:59,964 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard2-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:01:59,965 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: false, hasLeader : false 03:01:59,965 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config50 is no longer the leader 03:01:59,965 PM [test-akka.actor.default-dispatcher-16] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from null to Follower 03:01:59,965 PM [test-shard-dispatcher-18] [INFO] Shard - Shard created : member-1-shard-shard1-config50, persistent : true 03:01:59,965 PM [test-shard-dispatcher-18] [INFO] Shard - Starting recovery for member-1-shard-shard1-config50 with journal batch size 1 03:01:59,966 PM [test-akka.actor.default-dispatcher-15] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$l/member-1-shard-shard1-config50/member-1-shard-shard1-config50-notifier#1094726759 created and ready for shard:member-1-shard-shard1-config50 03:01:59,966 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$l/member-1-shard-shard1-config50/shard-member-1:snapshot-read#-1126044309] 03:01:59,967 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [shard2] 03:01:59,967 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config50 is initialized 03:01:59,967 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : shard1 03:01:59,967 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config50: Scheduling 3000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-shard1-config50, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 03:01:59,967 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config50 (Follower): Received TimeoutNow - switching to Candidate 03:01:59,971 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config50 (Candidate): Election: Candidate has following voting peers: [] 03:01:59,971 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config50: Set currentTerm=1, votedFor=member-1-shard-shard2-config50 03:01:59,974 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard2-config50 (Candidate): Starting new election term 1 03:01:59,974 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard2-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 03:01:59,974 PM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-shard2-config50 (Follower): Closing 03:01:59,974 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: false, hasLeader : false 03:01:59,974 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config50 is no longer the leader 03:01:59,974 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from Follower to Candidate 03:01:59,983 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 03:01:59,983 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:01:59,983 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [astronauts] 03:01:59,983 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config50 is initialized 03:01:59,983 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: false, hasLeader : false 03:01:59,983 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config50 is no longer the leader 03:01:59,983 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from null to Follower 03:01:59,983 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50 (Follower): Received TimeoutNow - switching to Candidate 03:01:59,984 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50 (Candidate): Election: Candidate has following voting peers: [] 03:01:59,984 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: Set currentTerm=1, votedFor=member-1-shard-astronauts-config50 03:01:59,984 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config50 (Candidate): Starting new election term 1 03:01:59,984 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 03:01:59,984 PM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-astronauts-config50 (Follower): Closing 03:01:59,984 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: false, hasLeader : false 03:01:59,984 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config50 is no longer the leader 03:01:59,984 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 03:01:59,984 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-shard1-config50: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:01:59,984 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: false, hasLeader : false 03:01:59,984 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config50 is no longer the leader 03:01:59,984 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50 (Follower): Received TimeoutNow - switching to Candidate 03:01:59,984 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50 (Candidate): Election: Candidate has following voting peers: [] 03:01:59,984 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50: Set currentTerm=1, votedFor=member-1-shard-shard1-config50 03:01:59,984 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-shard1-config50 (Candidate): Starting new election term 1 03:01:59,984 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-shard1-config50 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 03:01:59,984 PM [test-shard-dispatcher-21] [DEBUG] MessageAssembler - member-1-shard-shard1-config50 (Follower): Closing 03:01:59,984 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: false, hasLeader : false 03:01:59,984 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config50 is no longer the leader 03:01:59,984 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config50: Initializing shard [shard1] 03:01:59,984 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config50 is initialized 03:01:59,984 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config50 is initialized - notifying 1 OnShardInitialized callbacks 03:01:59,984 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from null to Follower 03:01:59,984 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from Follower to Candidate 03:01:59,984 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from Follower to Candidate 03:01:59,987 PM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : shard2 03:01:59,987 PM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config50: findLocalShard : astronauts 03:01:59,988 PM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManager - shard-manager-config50: onGetSnapshot 03:01:59,989 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Expecting 3 shard snapshot replies 03:01:59,993 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50 (Candidate): Received ElectionTimeout 03:01:59,993 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50 (Candidate): Received ElectionTimeout 03:01:59,993 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50 (Candidate): Received ElectionTimeout 03:02:00,0 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50 (Leader): Election: Leader has following peers: [] 03:02:00,0 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 03:02:00,0 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config50: onLeadershipChange, isLeader: true, hasLeader : true 03:02:00,0 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50 (Leader): Election: Leader has following peers: [] 03:02:00,0 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-shard1-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 03:02:00,0 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config50: onLeadershipChange, isLeader: true, hasLeader : true 03:02:00,0 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50 (Leader): Election: Leader has following peers: [] 03:02:00,0 PM [test-shard-dispatcher-18] [INFO] Shard - member-1-shard-shard2-config50 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 03:02:00,0 PM [test-shard-dispatcher-18] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config50: onLeadershipChange, isLeader: true, hasLeader : true 03:02:00,2 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50: became leader with frontend state for [] 03:02:00,2 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: became leader with frontend state for [] 03:02:00,2 PM [test-akka.actor.default-dispatcher-15] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , received role change from Candidate to Leader 03:02:00,2 PM [test-akka.actor.default-dispatcher-15] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , received role change from Candidate to Leader 03:02:00,6 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: onGetSnapshot 03:02:00,6 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50: onGetSnapshot 03:02:00,6 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 03:02:00,6 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 03:02:00,6 PM [test-akka.actor.default-dispatcher-15] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config50 , registered listener akka://test/user/$$l 03:02:00,6 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard1-config50, leaderId=member-1-shard-shard1-config50, leaderPayloadVersion=13] 03:02:00,6 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-shard1-config50 from Candidate to Leader 03:02:00,6 PM [test-akka.actor.default-dispatcher-15] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config50 , registered listener akka://test/user/$$l 03:02:00,6 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-astronauts-config50, leaderId=member-1-shard-astronauts-config50, leaderPayloadVersion=13] 03:02:00,6 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-astronauts-config50 from Candidate to Leader 03:02:00,10 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50: became leader with frontend state for [] 03:02:00,10 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50: onGetSnapshot 03:02:00,10 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 03:02:00,10 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , received role change from Candidate to Leader 03:02:00,10 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config50 , registered listener akka://test/user/$$l 03:02:00,10 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard2-config50, leaderId=member-1-shard-shard2-config50, leaderPayloadVersion=13] 03:02:00,10 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: Received role changed for member-1-shard-shard2-config50 from Candidate to Leader 03:02:00,10 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config50: All Shards are ready - data store config50 is ready 03:02:00,11 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 03:02:00,12 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-astronauts-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@1455bdcd, electionTerm=1, electionVotedFor=member-1-shard-astronauts-config50, ServerConfigPayload=null]] 03:02:00,12 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-shard2-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 03:02:00,12 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-shard2-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@2694846d, electionTerm=1, electionVotedFor=member-1-shard-shard2-config50, ServerConfigPayload=null]] 03:02:00,13 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config50: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 03:02:00,15 PM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: Received GetSnapshotReply [id=member-1-shard-shard1-config50, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@1a6fb315, electionTerm=1, electionVotedFor=member-1-shard-shard1-config50, ServerConfigPayload=null]] 03:02:00,15 PM [test-akka.actor.default-dispatcher-23] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config50: All shard snapshots received 03:02:00,16 PM [main] [INFO] ShardManagerTest - testRestoreFromSnapshot ending 03:02:00,118 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$l] 03:02:00,130 PM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-shard2-config50 03:02:00,130 PM [test-shard-dispatcher-19] [DEBUG] MessageSlicer - member-1-shard-shard2-config50 (Leader)_slicer-id-8: Closing 03:02:00,130 PM [test-shard-dispatcher-18] [INFO] Shard - Stopping Shard member-1-shard-astronauts-config50 03:02:00,130 PM [test-shard-dispatcher-18] [DEBUG] MessageSlicer - member-1-shard-astronauts-config50 (Leader)_slicer-id-6: Closing 03:02:00,130 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config50: Aborting 0 pending queued transactions 03:02:00,130 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config50: Aborting 0 pending queued transactions 03:02:00,130 PM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-shard1-config50 03:02:00,130 PM [test-shard-dispatcher-19] [DEBUG] MessageSlicer - member-1-shard-shard1-config50 (Leader)_slicer-id-7: Closing 03:02:00,130 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config50: Aborting 0 pending queued transactions 03:02:00,130 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Stopping ShardManager shard-manager-config50 03:02:00,133 PM [main] [INFO] ShardManagerTest - testOnCreateShardWithNoInitialSchemaContext starting 03:02:00,137 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Starting ShardManager shard-manager-config51 03:02:00,138 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config51 03:02:00,148 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config51: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@158ad959] 03:02:00,148 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config51 doCreateShard: shardId: member-1-shard-foo-config51, memberNames: [MemberName{name=member-1}], peerAddresses: {}, isActiveMember: true 03:02:00,148 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:02:00,148 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - Creating Shard member-1-shard-foo-config51 03:02:00,149 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config51: findLocalShard : foo 03:02:00,149 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config51: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config51, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 03:02:00,150 PM [test-shard-dispatcher-21] [INFO] Shard - Shard created : member-1-shard-foo-config51, persistent : true 03:02:00,151 PM [test-shard-dispatcher-21] [INFO] Shard - Starting recovery for member-1-shard-foo-config51 with journal batch size 1 03:02:00,151 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$B/member-1-shard-foo-config51/shard-member-1:snapshot-read#17070491] 03:02:00,151 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$B/member-1-shard-foo-config51/member-1-shard-foo-config51-notifier#-504682501 created and ready for shard:member-1-shard-foo-config51 03:02:00,152 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config51: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 03:02:00,152 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-foo-config51: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:02:00,152 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config51: Initializing shard [foo] 03:02:00,152 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: false, hasLeader : false 03:02:00,152 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-foo-config51 is initialized 03:02:00,153 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config51: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config51 is no longer the leader 03:02:00,153 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardInformation - Shard member-1-shard-foo-config51 is initialized - notifying 1 OnShardInitialized callbacks 03:02:00,153 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config51 (Follower): Received TimeoutNow - switching to Candidate 03:02:00,153 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config51 (Candidate): Election: Candidate has following voting peers: [] 03:02:00,153 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config51: Set currentTerm=1, votedFor=member-1-shard-foo-config51 03:02:00,153 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from null to Follower 03:02:00,153 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-foo-config51 (Candidate): Starting new election term 1 03:02:00,153 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-foo-config51 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 03:02:00,153 PM [test-shard-dispatcher-21] [DEBUG] MessageAssembler - member-1-shard-foo-config51 (Follower): Closing 03:02:00,153 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: false, hasLeader : false 03:02:00,153 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config51: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config51 is no longer the leader 03:02:00,153 PM [main] [INFO] ShardManagerTest - testOnCreateShardWithNoInitialSchemaContext ending 03:02:00,153 PM [test-akka.actor.default-dispatcher-23] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from Follower to Candidate 03:02:00,154 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config51 (Candidate): Received ElectionTimeout 03:02:00,154 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config51 (Leader): Election: Leader has following peers: [] 03:02:00,154 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-foo-config51 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 03:02:00,154 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config51: onLeadershipChange, isLeader: true, hasLeader : true 03:02:00,154 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config51: became leader with frontend state for [] 03:02:00,154 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , received role change from Candidate to Leader 03:02:00,154 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config51 , registered listener akka://test/user/$B 03:02:00,155 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config51: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-foo-config51, leaderId=member-1-shard-foo-config51, leaderPayloadVersion=13] 03:02:00,155 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config51: Received role changed for member-1-shard-foo-config51 from Candidate to Leader 03:02:00,155 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - shard-manager-config51: All Shards are ready - data store config51 is ready 03:02:00,255 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$B#-192536566] 03:02:00,267 PM [test-shard-dispatcher-21] [INFO] Shard - Stopping Shard member-1-shard-foo-config51 03:02:00,267 PM [test-shard-dispatcher-21] [DEBUG] MessageSlicer - member-1-shard-foo-config51 (Leader)_slicer-id-10: Closing 03:02:00,267 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config51: Aborting 0 pending queued transactions 03:02:00,267 PM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Stopping ShardManager shard-manager-config51 03:02:00,270 PM [main] [INFO] ShardManagerTest - testGetSnapshot starting 03:02:00,273 PM [test-akka.actor.default-dispatcher-5] [INFO] ShardManager - Starting ShardManager shard-manager-config52 03:02:00,274 PM [test-akka.actor.default-dispatcher-10] [INFO] ShardManager - Recovery complete : shard-manager-config52 03:02:00,274 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config52: Creating local shard: member-1-shard-shard2-config52 03:02:00,275 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManager - shard-manager-config52: Creating local shard: member-1-shard-shard1-config52 03:02:00,284 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 03:02:00,284 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:02:00,284 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-shard2-config52 03:02:00,285 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - Creating Shard member-1-shard-shard1-config52 03:02:00,285 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : shard1 03:02:00,286 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardManager - shard-manager-config52: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-shard1-config52, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 03:02:00,286 PM [test-shard-dispatcher-21] [INFO] Shard - Shard created : member-1-shard-shard2-config52, persistent : true 03:02:00,286 PM [test-shard-dispatcher-19] [INFO] Shard - Shard created : member-1-shard-shard1-config52, persistent : true 03:02:00,287 PM [test-shard-dispatcher-21] [INFO] Shard - Starting recovery for member-1-shard-shard2-config52 with journal batch size 1 03:02:00,287 PM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-shard2-config52/member-1-shard-shard2-config52-notifier#-698362606 created and ready for shard:member-1-shard-shard2-config52 03:02:00,287 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-shard2-config52/shard-member-1:snapshot-read#1818349187] 03:02:00,288 PM [test-akka.actor.default-dispatcher-5] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-shard1-config52/member-1-shard-shard1-config52-notifier#926676298 created and ready for shard:member-1-shard-shard1-config52 03:02:00,288 PM [test-akka.actor.default-dispatcher-5] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-shard1-config52/shard-member-1:snapshot-read#-1483104758] 03:02:00,288 PM [test-shard-dispatcher-19] [INFO] Shard - Starting recovery for member-1-shard-shard1-config52 with journal batch size 1 03:02:00,290 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 03:02:00,290 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 03:02:00,290 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-shard2-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:02:00,290 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard1-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:02:00,290 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: false, hasLeader : false 03:02:00,290 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config52 is no longer the leader 03:02:00,290 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: false, hasLeader : false 03:02:00,290 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config52 is no longer the leader 03:02:00,290 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52 (Follower): Received TimeoutNow - switching to Candidate 03:02:00,291 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52 (Candidate): Election: Candidate has following voting peers: [] 03:02:00,291 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: Set currentTerm=1, votedFor=member-1-shard-shard1-config52 03:02:00,291 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52 (Follower): Received TimeoutNow - switching to Candidate 03:02:00,291 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard1-config52 (Candidate): Starting new election term 1 03:02:00,291 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard1-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 03:02:00,291 PM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-shard1-config52 (Follower): Closing 03:02:00,291 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: false, hasLeader : false 03:02:00,291 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52 (Candidate): Election: Candidate has following voting peers: [] 03:02:00,291 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard1-config52 is no longer the leader 03:02:00,291 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: Set currentTerm=1, votedFor=member-1-shard-shard2-config52 03:02:00,291 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [shard2] 03:02:00,291 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-shard2-config52 (Candidate): Starting new election term 1 03:02:00,291 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard2-config52 is initialized 03:02:00,291 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [shard1] 03:02:00,291 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-shard2-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 03:02:00,291 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config52 is initialized 03:02:00,291 PM [test-shard-dispatcher-21] [DEBUG] MessageAssembler - member-1-shard-shard2-config52 (Follower): Closing 03:02:00,291 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-shard1-config52 is initialized - notifying 1 OnShardInitialized callbacks 03:02:00,291 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: false, hasLeader : false 03:02:00,291 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-shard2-config52 is no longer the leader 03:02:00,291 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from null to Follower 03:02:00,291 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from Follower to Candidate 03:02:00,291 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from null to Follower 03:02:00,291 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from Follower to Candidate 03:02:00,292 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : shard2 03:02:00,292 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 03:02:00,293 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52 (Candidate): Received ElectionTimeout 03:02:00,293 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config52 (Candidate): Received ElectionTimeout 03:02:00,293 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52 (Leader): Election: Leader has following peers: [] 03:02:00,293 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-shard2-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 03:02:00,293 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard2-config52: onLeadershipChange, isLeader: true, hasLeader : true 03:02:00,293 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard2-config52: became leader with frontend state for [] 03:02:00,293 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config52 (Leader): Election: Leader has following peers: [] 03:02:00,293 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-shard1-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 03:02:00,293 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , received role change from Candidate to Leader 03:02:00,293 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-shard1-config52: onLeadershipChange, isLeader: true, hasLeader : true 03:02:00,293 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard2-config52 , registered listener akka://test/user/$$m 03:02:00,293 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard1-config52: became leader with frontend state for [] 03:02:00,293 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , received role change from Candidate to Leader 03:02:00,293 PM [test-akka.actor.default-dispatcher-10] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-shard1-config52 , registered listener akka://test/user/$$m 03:02:00,293 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Expecting 2 shard snapshot replies 03:02:00,294 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard2-config52, leaderId=member-1-shard-shard2-config52, leaderPayloadVersion=13] 03:02:00,294 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-shard2-config52 from Candidate to Leader 03:02:00,294 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-shard1-config52, leaderId=member-1-shard-shard1-config52, leaderPayloadVersion=13] 03:02:00,294 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-shard1-config52 from Candidate to Leader 03:02:00,294 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - shard-manager-config52: All Shards are ready - data store config52 is ready 03:02:00,294 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: onGetSnapshot 03:02:00,294 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: onGetSnapshot 03:02:00,294 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 03:02:00,294 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 03:02:00,295 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 03:02:00,295 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 03:02:00,295 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard1-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@310778d3, electionTerm=1, electionVotedFor=member-1-shard-shard1-config52, ServerConfigPayload=null]] 03:02:00,296 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard2-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@57d8c355, electionTerm=1, electionVotedFor=member-1-shard-shard2-config52, ServerConfigPayload=null]] 03:02:00,296 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: All shard snapshots received 03:02:00,297 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: onAddShardReplica: AddShardReplica[ShardName=astronauts] 03:02:00,298 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config52: Sending AddServer message to peer akka://test@127.0.0.1:2565/system/testProbe-100#-264581445 for shard member-1-shard-astronauts-config52 03:02:00,299 PM [test-shard-dispatcher-19] [INFO] Shard - Shard created : member-1-shard-astronauts-config52, persistent : true 03:02:00,300 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$$m/member-1-shard-astronauts-config52/member-1-shard-astronauts-config52-notifier#953420232 created and ready for shard:member-1-shard-astronauts-config52 03:02:00,300 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$$m/member-1-shard-astronauts-config52/shard-member-1:snapshot-read#-184002544] 03:02:00,300 PM [test-shard-dispatcher-19] [INFO] Shard - Starting recovery for member-1-shard-astronauts-config52 with journal batch size 1 03:02:00,300 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Received AddServerReply{status=OK, leaderHint=} for shard astronauts from leader akka://test@127.0.0.1:2565/system/testProbe-100#-264581445 03:02:00,300 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Leader shard successfully added the replica shard astronauts 03:02:00,301 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Sending new DatastoreContext to member-1-shard-astronauts-config52 03:02:00,301 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: persisting the shard list [shard2, astronauts, shard1] 03:02:00,301 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: findLocalShard : astronauts 03:02:00,301 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Scheduling 600 ms timer to wait for shard ShardInformation [shardId=member-1-shard-astronauts-config52, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 03:02:00,301 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52 saved ShardManager snapshot successfully. Deleting the prev snapshot if available 03:02:00,301 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Successfully deleted prior snapshots 03:02:00,302 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 03:02:00,303 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config52: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 03:02:00,303 PM [test-shard-dispatcher-19] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: RaftPolicy used with prev.config org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy, RaftPolicy used with newConfig null 03:02:00,303 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManager - shard-manager-config52: Initializing shard [astronauts] 03:02:00,303 PM [test-akka.actor.default-dispatcher-11] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from null to Follower 03:02:00,303 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config52 is initialized 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: Re-initializing to Follower with previous leaderId null 03:02:00,303 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardInformation - Shard member-1-shard-astronauts-config52 is initialized - notifying 1 OnShardInitialized callbacks 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-astronauts-config52 (Follower): Closing 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52 (Follower): Received TimeoutNow - switching to Candidate 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52 (Candidate): Election: Candidate has following voting peers: [] 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: Set currentTerm=1, votedFor=member-1-shard-astronauts-config52 03:02:00,303 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config52 (Candidate): Starting new election term 1 03:02:00,303 PM [test-shard-dispatcher-19] [INFO] Shard - member-1-shard-astronauts-config52 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] MessageAssembler - member-1-shard-astronauts-config52 (Follower): Closing 03:02:00,303 PM [test-shard-dispatcher-19] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: false, hasLeader : false 03:02:00,304 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onStateChanged: Closing all transaction chains because shard member-1-shard-astronauts-config52 is no longer the leader 03:02:00,304 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from Follower to Candidate 03:02:00,304 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config52: onGetSnapshot 03:02:00,305 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-astronauts-config52 (Candidate): Received ElectionTimeout 03:02:00,305 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-astronauts-config52 (Leader): Election: Leader has following peers: [] 03:02:00,305 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-astronauts-config52 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 03:02:00,305 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-astronauts-config52: onLeadershipChange, isLeader: true, hasLeader : true 03:02:00,305 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-astronauts-config52: became leader with frontend state for [] 03:02:00,305 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , received role change from Candidate to Leader 03:02:00,305 PM [test-akka.actor.default-dispatcher-14] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-astronauts-config52 , registered listener akka://test/user/$$m 03:02:00,306 PM [test-akka.actor.default-dispatcher-14] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Expecting 3 shard snapshot replies 03:02:00,306 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config52: Received LeaderStateChanged message: LeaderStateChanged [memberId=member-1-shard-astronauts-config52, leaderId=member-1-shard-astronauts-config52, leaderPayloadVersion=13] 03:02:00,306 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config52: Received role changed for member-1-shard-astronauts-config52 from Candidate to Leader 03:02:00,306 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config52: All Shards are ready - data store config52 is ready 03:02:00,306 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: onGetSnapshot 03:02:00,306 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: onGetSnapshot 03:02:00,306 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 03:02:00,306 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 03:02:00,306 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 03:02:00,306 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-astronauts-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 03:02:00,307 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: onGetSnapshot 03:02:00,307 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 03:02:00,307 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-astronauts-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@17208fd6, electionTerm=1, electionVotedFor=member-1-shard-astronauts-config52, ServerConfigPayload=null]] 03:02:00,307 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: requesting serialization of snapshot MetadataShardDataTreeSnapshot{metadata={class org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata=FrontendShardDataTreeSnapshotMetadata{clients=[]}}} 03:02:00,307 PM [test-akka.actor.default-dispatcher-11] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard2-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@186e2b30, electionTerm=1, electionVotedFor=member-1-shard-shard2-config52, ServerConfigPayload=null]] 03:02:00,307 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: Received GetSnapshotReply [id=member-1-shard-shard1-config52, snapshot=Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=org.opendaylight.controller.cluster.datastore.persisted.ShardSnapshotState@20ce80b4, electionTerm=1, electionVotedFor=member-1-shard-shard1-config52, ServerConfigPayload=null]] 03:02:00,307 PM [test-akka.actor.default-dispatcher-10] [DEBUG] ShardManagerGetSnapshotReplyActor - shard-manager-config52: All shard snapshots received 03:02:00,308 PM [main] [INFO] ShardManagerTest - testGetSnapshot ending 03:02:00,410 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$m] 03:02:00,422 PM [test-shard-dispatcher-18] [INFO] Shard - Stopping Shard member-1-shard-astronauts-config52 03:02:00,422 PM [test-shard-dispatcher-19] [INFO] Shard - Stopping Shard member-1-shard-shard1-config52 03:02:00,422 PM [test-shard-dispatcher-18] [DEBUG] MessageSlicer - member-1-shard-astronauts-config52 (Leader)_slicer-id-16: Closing 03:02:00,422 PM [test-shard-dispatcher-19] [DEBUG] MessageSlicer - member-1-shard-shard1-config52 (Leader)_slicer-id-14: Closing 03:02:00,422 PM [test-shard-dispatcher-19] [DEBUG] Shard - member-1-shard-shard1-config52: Aborting 0 pending queued transactions 03:02:00,422 PM [test-shard-dispatcher-18] [DEBUG] Shard - member-1-shard-astronauts-config52: Aborting 0 pending queued transactions 03:02:00,422 PM [test-shard-dispatcher-21] [INFO] Shard - Stopping Shard member-1-shard-shard2-config52 03:02:00,422 PM [test-shard-dispatcher-21] [DEBUG] MessageSlicer - member-1-shard-shard2-config52 (Leader)_slicer-id-13: Closing 03:02:00,422 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-shard2-config52: Aborting 0 pending queued transactions 03:02:00,424 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Stopping ShardManager shard-manager-config52 03:02:00,426 PM [main] [INFO] ShardManagerTest - testWhenMultipleShardsPresentSyncStatusMustBeTrueForAllShards starting 03:02:00,431 PM [main] [INFO] ShardManager - Starting ShardManager shard-manager-config53 03:02:00,432 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Recovery complete : shard-manager-config53 03:02:00,433 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config53: Creating local shard: member-1-shard-default-config53 03:02:00,433 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config53: Creating local shard: member-1-shard-astronauts-config53 03:02:00,437 PM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-default-config53 from Follower to Leader 03:02:00,438 PM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-astronauts-config53 from Follower to Leader 03:02:00,438 PM [main] [INFO] ShardManager - shard-manager-config53: Received role changed for member-1-shard-astronauts-config53 from Leader to Follower 03:02:00,438 PM [main] [INFO] ShardManager - shard-manager-config53 Received follower initial sync status for member-1-shard-astronauts-config53 status sync done true 03:02:00,438 PM [main] [INFO] ShardManagerTest - testWhenMultipleShardsPresentSyncStatusMustBeTrueForAllShards ending 03:02:00,539 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/$$n] 03:02:00,550 PM [main] [INFO] ShardManager - Stopping ShardManager shard-manager-config53 03:02:00,553 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithFindPrimaryTimeout starting 03:02:00,561 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - Starting ShardManager shard-manager-config54 03:02:00,563 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Recovery complete : shard-manager-config54 03:02:00,571 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:02:00,571 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config54: Received MemberUp: memberName: MemberName{name=member-2}, address: akka://non-existent@127.0.0.1:5 03:02:00,572 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config54: All Shards are ready - data store config54 is ready 03:02:00,572 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config54: onAddShardReplica: AddShardReplica[ShardName=astronauts] 03:02:00,572 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config54: In findPrimary: org.opendaylight.controller.cluster.datastore.messages.FindPrimary [shardName=astronauts, waitUntilReady=true] 03:02:00,572 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config54: findPrimary for astronauts forwarding to remote ShardManager akka://non-existent@127.0.0.1:5/user/shardmanager-config54, visitedAddresses: [akka://test@127.0.0.1:2550/user/shardmanager-config54] 03:02:00,597 PM [test-akka.actor.default-dispatcher-16] [WARN] Materializer - [outbound connection to [akka://non-existent@127.0.0.1:5], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1/:5,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused 03:02:00,598 PM [test-akka.actor.default-dispatcher-16] [WARN] Materializer - [outbound connection to [akka://non-existent@127.0.0.1:5], control stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(127.0.0.1/:5,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused 03:02:00,791 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config54: Received failure from FindPrimary for shard astronauts akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://test/user/shardmanager-config54#772541788]] after [200 ms]. Message of type [org.opendaylight.controller.cluster.datastore.messages.FindPrimary]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply. 03:02:00,792 PM [main] [INFO] ShardManagerTest - testAddShardReplicaWithFindPrimaryTimeout ending 03:02:00,893 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/shardmanager-config54#772541788] 03:02:00,904 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - Stopping ShardManager shard-manager-config54 03:02:00,907 PM [main] [INFO] ShardManagerTest - testOnCreateShard starting 03:02:00,911 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - Starting ShardManager shard-manager-config55 03:02:00,912 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Recovery complete : shard-manager-config55 03:02:00,913 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - Got updated SchemaContext: # of modules 1 03:02:00,922 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config55: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@1256371b] 03:02:00,922 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config55 doCreateShard: shardId: member-1-shard-foo-config55, memberNames: [MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], peerAddresses: {member-6-shard-foo-config55=null, member-5-shard-foo-config55=null}, isActiveMember: true 03:02:00,924 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config55: findLocalShard : foo 03:02:00,924 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardManager - shard-manager-config55: Scheduling 300000 ms timer to wait for shard ShardInformation [shardId=member-1-shard-foo-config55, leaderAvailable=false, actorInitialized=false, followerSyncStatus=false, role=null, leaderId=null, activeMember=true] 03:02:00,928 PM [test-shard-dispatcher-21] [INFO] Shard - Shard created : member-1-shard-foo-config55, persistent : false 03:02:00,929 PM [test-akka.actor.default-dispatcher-22] [INFO] RoleChangeNotifier - RoleChangeNotifier:akka://test@127.0.0.1:2565/user/$C/member-1-shard-foo-config55/member-1-shard-foo-config55-notifier#-296951161 created and ready for shard:member-1-shard-foo-config55 03:02:00,929 PM [test-akka.actor.default-dispatcher-22] [DEBUG] ShardSnapshotActor - Actor created Actor[akka://test/user/$C/member-1-shard-foo-config55/shard-member-1:snapshot-read#1400762758] 03:02:00,930 PM [test-shard-dispatcher-21] [INFO] Shard - Starting recovery for member-1-shard-foo-config55 with journal batch size 1 03:02:00,936 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config55: onReceiveRecover: Received message class akka.persistence.RecoveryCompleted$ from Actor[akka://test/deadLetters] 03:02:00,936 PM [test-shard-dispatcher-21] [INFO] Shard - member-1-shard-foo-config55: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:02:00,937 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardManager - shard-manager-config55: Initializing shard [foo] 03:02:00,937 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-foo-config55 is initialized 03:02:00,937 PM [test-akka.actor.default-dispatcher-15] [DEBUG] ShardInformation - Shard member-1-shard-foo-config55 is initialized - notifying 1 OnShardInitialized callbacks 03:02:00,937 PM [test-shard-dispatcher-21] [DEBUG] DataTreeChangeListenerSupport - member-1-shard-foo-config55: onLeadershipChange, isLeader: false, hasLeader : false 03:02:00,937 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config55: onStateChanged: Closing all transaction chains because shard member-1-shard-foo-config55 is no longer the leader 03:02:00,937 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config55: onCreateShard: CreateShard [moduleShardConfig=ModuleShardConfiguration [namespace=foo-ns, moduleName=foo-module, shardName=foo, shardMemberNames=[MemberName{name=member-1}, MemberName{name=member-5}, MemberName{name=member-6}], shardStrategyName=null], shardPropsCreator=org.opendaylight.controller.cluster.datastore.Shard$Builder@1256371b] 03:02:00,937 PM [test-akka.actor.default-dispatcher-16] [DEBUG] ShardManager - shard-manager-config55: Shard foo already exists 03:02:00,937 PM [test-akka.actor.default-dispatcher-16] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config55 , received role change from null to Follower 03:02:00,937 PM [test-akka.actor.default-dispatcher-16] [INFO] RoleChangeNotifier - RoleChangeNotifier for member-1-shard-foo-config55 , registered listener akka://test/user/$C 03:02:00,937 PM [test-akka.actor.default-dispatcher-16] [INFO] ShardManager - shard-manager-config55: Received role changed for member-1-shard-foo-config55 from null to Follower 03:02:00,937 PM [main] [INFO] ShardManagerTest - testOnCreateShard ending 03:02:01,39 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$C#-1288471398] 03:02:01,51 PM [test-shard-dispatcher-21] [INFO] Shard - Stopping Shard member-1-shard-foo-config55 03:02:01,51 PM [test-shard-dispatcher-21] [DEBUG] MessageAssembler - member-1-shard-foo-config55 (Follower): Closing 03:02:01,51 PM [test-shard-dispatcher-21] [DEBUG] Shard - member-1-shard-foo-config55: Aborting 0 pending queued transactions 03:02:01,52 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Stopping ShardManager shard-manager-config55 03:02:01,54 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config1 03:02:01,54 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Stopping ShardManager shard-manager-config35 03:02:01,54 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config43 03:02:01,54 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Stopping ShardManager shard-manager-config11 03:02:01,54 PM [test-akka.actor.default-dispatcher-15] [INFO] ShardManager - Stopping ShardManager shard-manager-config21 03:02:01,54 PM [test-akka.actor.default-dispatcher-22] [INFO] ShardManager - Stopping ShardManager shard-manager-config46 03:02:01,55 PM [test-akka.actor.default-dispatcher-16] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:02:01,55 PM [test-akka.actor.default-dispatcher-16] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 03:02:01,59 PM [test-akka.actor.default-dispatcher-16] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [WARN] [03/07/2024 15:02:01.062] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Could not unregister Cluster JMX MBean with name=akka:type=Cluster as it was not found. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config