[DEBUG] [03/15/2024 12:30:14.708] [main] [EventStream(akka://test)] logger log1-TestEventListener started 12:30:14,708 PM [test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [03/15/2024 12:30:14.708] [main] [EventStream(akka://test)] logger log2-Slf4jLogger started [DEBUG] [03/15/2024 12:30:14.708] [main] [EventStream(akka://test)] Default Loggers started [INFO] [03/15/2024 12:30:14.721] [main] [ArteryTransport(akka://test)] Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.170.99:25520] with UID [-8280764394726618646] 12:30:14,722 PM [test-akka.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.170.99:25520] with UID [-8280764394726618646] [INFO] [03/15/2024 12:30:14.723] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.170.99:25520] - Starting up, Akka version [2.6.21] ... 12:30:14,723 PM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@10.30.170.99:25520] - Starting up, Akka version [2.6.21] ... [INFO] [03/15/2024 12:30:14.731] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.170.99:25520] - Registered cluster JMX MBean [akka:type=Cluster] [INFO] [03/15/2024 12:30:14.731] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.170.99:25520] - Started up successfully 12:30:14,731 PM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.170.99:25520] - Registered cluster JMX MBean [akka:type=Cluster] 12:30:14,731 PM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.170.99:25520] - Started up successfully [INFO] [03/15/2024 12:30:14.737] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.170.99:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing [INFO] [03/15/2024 12:30:14.737] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.170.99:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:30:14,738 PM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.170.99:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 12:30:14,738 PM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.170.99:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 12:30:14,745 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 12:30:14,745 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/15/2024 12:30:14.753] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower656]: org.opendaylight.controller.cluster.common.actor.Monitor@67cb20cb [INFO] [akkaDeadLetter][03/15/2024 12:30:14.753] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower656] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:14,754 PM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower656] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:14,755 PM [test-akka.actor.default-dispatcher-10] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower656 with journal batch size 1000 12:30:14,757 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656: SnapshotOffer called. 12:30:14,757 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656: Moving last applied index from -1 to -1 12:30:14,757 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656: Set currentTerm=1, votedFor=null 12:30:14,757 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656: Updated server config: isVoting: true, peers: [PeerInfo [id=follower658, address=akka://test/user/follower658, votingState=NON_VOTING], PeerInfo [id=follower657, address=akka://test/user/follower657, votingState=VOTING], PeerInfo [id=leader655, address=akka://test/user/leader655, votingState=VOTING]] 12:30:14,757 PM [test-akka.actor.default-dispatcher-10] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower656 in 110.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:14,758 PM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower656: Recovery completed in 1.249 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:14,759 PM [test-akka.actor.default-dispatcher-10] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower657 with journal batch size 1000 12:30:14,759 PM [test-akka.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower657] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:14,761 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower658 with journal batch size 1000 12:30:14,763 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader655 with journal batch size 1000 [WARN] [03/15/2024 12:30:14.765] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower657]: org.opendaylight.controller.cluster.common.actor.Monitor@5214df90 [INFO] [akkaDeadLetter][03/15/2024 12:30:14.759] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower657] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [03/15/2024 12:30:14.765] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower658]: org.opendaylight.controller.cluster.common.actor.Monitor@5cd33ba2 [WARN] [03/15/2024 12:30:14.765] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader655]: org.opendaylight.controller.cluster.common.actor.Monitor@58ecb4e0 12:30:14,768 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657: SnapshotOffer called. 12:30:14,768 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657: Moving last applied index from -1 to -1 12:30:14,768 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657: Set currentTerm=1, votedFor=null 12:30:14,768 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657: Updated server config: isVoting: true, peers: [PeerInfo [id=follower656, address=akka://test/user/follower656, votingState=VOTING], PeerInfo [id=follower658, address=akka://test/user/follower658, votingState=NON_VOTING], PeerInfo [id=leader655, address=akka://test/user/leader655, votingState=VOTING]] 12:30:14,768 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower657 in 88.26 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:14,768 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: SnapshotOffer called. 12:30:14,768 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Moving last applied index from -1 to -1 12:30:14,768 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Set currentTerm=1, votedFor=null 12:30:14,768 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Updated server config: isVoting: false, peers: [PeerInfo [id=follower656, address=akka://test/user/follower656, votingState=VOTING], PeerInfo [id=follower657, address=akka://test/user/follower657, votingState=VOTING], PeerInfo [id=leader655, address=akka://test/user/leader655, votingState=VOTING]] 12:30:14,769 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower658 in 33.52 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:14,769 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655: SnapshotOffer called. 12:30:14,769 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655: Moving last applied index from -1 to -1 12:30:14,769 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655: Set currentTerm=1, votedFor=null 12:30:14,769 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655: Updated server config: isVoting: true, peers: [PeerInfo [id=follower656, address=akka://test/user/follower656, votingState=VOTING], PeerInfo [id=follower658, address=akka://test/user/follower658, votingState=NON_VOTING], PeerInfo [id=follower657, address=akka://test/user/follower657, votingState=VOTING]] 12:30:14,769 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader655 in 31.59 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:14,769 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower657: Recovery completed in 1.295 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:14,769 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Recovery completed in 843.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:14,770 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader655: Recovery completed in 1.178 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:14,774 PM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower658] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][03/15/2024 12:30:14.774] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower658] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:14,774 PM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader655] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][03/15/2024 12:30:14.774] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader655] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:15,163 PM [test-akka.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:30:15,164 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.170.99:25520, Removed) 12:30:15,164 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Follower): Received ElectionTimeout - switching to Candidate 12:30:15,164 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Candidate): Election: Candidate has following voting peers: [follower656, follower657] 12:30:15,164 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655: Set currentTerm=2, votedFor=leader655 12:30:15,164 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Candidate): Starting new election term 2 12:30:15,164 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Candidate): Sending RequestVote [term=2, candidateId=leader655, lastLogIndex=-1, lastLogTerm=-1] to peer follower656 12:30:15,164 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Candidate): Sending RequestVote [term=2, candidateId=leader655, lastLogIndex=-1, lastLogTerm=-1] to peer follower657 12:30:15,164 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:15,164 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656 (Follower): Cluster state: [] 12:30:15,164 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:30:15,164 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader655, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:15,164 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower656 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader655, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:15,164 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656: Set currentTerm=2, votedFor=null 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656 (Follower): In requestVote: RequestVote [term=2, candidateId=leader655, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656: Set currentTerm=2, votedFor=leader655 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657 (Follower): Cluster state: [] 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader655, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower657 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader655, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657: Set currentTerm=2, votedFor=null 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657 (Follower): In requestVote: RequestVote [term=2, candidateId=leader655, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657: Set currentTerm=2, votedFor=leader655 12:30:15,165 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:15,166 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:30:15,166 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:30:15,166 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Election: Leader has following peers: [follower656, follower658, follower657] 12:30:15,167 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Checking sendAppendEntries for follower follower656: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:15,167 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Checking sendAppendEntries for follower follower658: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:15,167 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Checking sendAppendEntries for follower follower657: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:15,167 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:30:15,167 PM [test-akka.actor.default-dispatcher-10] [DEBUG] SyncStatusTracker - follower656: Last sync leader does not match current leader leader655, need to catch up to -1 12:30:15,167 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower656 updated: matchIndex: -1, nextIndex: 0 12:30:15,167 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Checking sendAppendEntries for follower follower656: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:15,167 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): Term 2 in "AppendEntries [leaderId=leader655, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:30:15,167 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Set currentTerm=2, votedFor=null 12:30:15,167 PM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower657: Last sync leader does not match current leader leader655, need to catch up to -1 12:30:15,167 PM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower658: Last sync leader does not match current leader leader655, need to catch up to -1 12:30:15,167 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower657 updated: matchIndex: -1, nextIndex: 0 12:30:15,167 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Checking sendAppendEntries for follower follower657: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:15,167 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower658 updated: matchIndex: -1, nextIndex: 0 [WARN] [03/15/2024 12:30:15.167] [test-akka.actor.default-dispatcher-10] [akka://test/user/follower656] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@767fb82 12:30:15,167 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Checking sendAppendEntries for follower follower658: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/15/2024 12:30:15.167] [test-akka.actor.default-dispatcher-10] [akka://test/user/follower657] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@119b97e 12:30:15,168 PM [test-akka.actor.default-dispatcher-18] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower656] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][03/15/2024 12:30:15.167] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower656] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower656] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:15,168 PM [test-akka.actor.default-dispatcher-18] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower657] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][03/15/2024 12:30:15.167] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower657] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower657] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [03/15/2024 12:30:15.168] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower658] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@733d453f [INFO] [akkaDeadLetter][03/15/2024 12:30:15.168] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower658] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower658] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:15,168 PM [test-akka.actor.default-dispatcher-15] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower658] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:15,175 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:30:15,175 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader655 starting 12:30:15,177 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655: onShutDown 12:30:15,178 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655: Initiating leader transfer 12:30:15,178 PM [test-akka.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader655: pauseLeader successfully completed - doing transfer 12:30:15,178 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Attempting to transfer leadership 12:30:15,178 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower656: onLeaderTransitioning: LeaderTransitioning [leaderId=leader655] 12:30:15,178 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: onLeaderTransitioning: LeaderTransitioning [leaderId=leader655] 12:30:15,178 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Checking sendAppendEntries for follower follower656: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:15,178 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower657: onLeaderTransitioning: LeaderTransitioning [leaderId=leader655] 12:30:15,178 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Checking sendAppendEntries for follower follower658: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:15,178 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Checking sendAppendEntries for follower follower657: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:15,178 PM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower656: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) 12:30:15,178 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower658: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) 12:30:15,179 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower657: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) [WARN] [03/15/2024 12:30:15.179] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower656] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2112680d [WARN] [03/15/2024 12:30:15.179] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower658] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3c33d95f [INFO] [akkaDeadLetter][03/15/2024 12:30:15.179] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower658] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower658] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][03/15/2024 12:30:15.179] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower656] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower656] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:15,179 PM [test-akka.actor.default-dispatcher-17] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower658] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:15,179 PM [test-akka.actor.default-dispatcher-17] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower656] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [03/15/2024 12:30:15.179] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower657] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3979d697 12:30:15,180 PM [test-akka.actor.default-dispatcher-15] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower657] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][03/15/2024 12:30:15.179] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower657] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower657] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 12:30:15,284 PM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower657: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) 12:30:15,284 PM [test-akka.actor.default-dispatcher-10] [DEBUG] SyncStatusTracker - follower656: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) 12:30:15,284 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower658: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) 12:30:15,404 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower657: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) 12:30:15,404 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower658: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) 12:30:15,405 PM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower656: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) 12:30:15,523 PM [test-akka.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader655: leader transfer aborted 12:30:15,523 PM [test-akka.actor.default-dispatcher-12] [WARN] RaftActorLeadershipTransferCohort - leader655: Failed to transfer leadership in 344.9 ms 12:30:15,523 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655: leader transfer failed - sending PoisonPill 12:30:15,523 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader655 (Leader): Leadership transfer expired 12:30:15,524 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower656: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) 12:30:15,524 PM [test-akka.actor.default-dispatcher-10] [DEBUG] SyncStatusTracker - follower658: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) 12:30:15,524 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower657: Lagging 0 entries behind leader leader655 and reached -1 (of expected -1) [WARN] [03/15/2024 12:30:15.524] [test-akka.actor.default-dispatcher-15] [akka://test/user/leader655] received dead letter from TestActor[akka://test/user/follower656]: AppendEntriesReply [term=2, success=true, followerId=follower656, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [03/15/2024 12:30:15.525] [test-akka.actor.default-dispatcher-15] [akka://test/user/leader655] received dead letter from TestActor[akka://test/user/follower658]: AppendEntriesReply [term=2, success=true, followerId=follower658, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [03/15/2024 12:30:15.525] [test-akka.actor.default-dispatcher-15] [akka://test/user/leader655] received dead letter from TestActor[akka://test/user/follower657]: AppendEntriesReply [term=2, success=true, followerId=follower657, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:30:15,526 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader655 ending 12:30:15,527 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 12:30:15,527 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower656-notifier659#-646817536] 12:30:15,540 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower656-collector660#-461295445] 12:30:15,540 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower656] 12:30:15,540 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower657-notifier661#1518384884] 12:30:15,540 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower657-collector662#-2027328836] 12:30:15,541 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower657] 12:30:15,541 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower658-notifier663#-1961585379] 12:30:15,541 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower658-collector664#89231918] 12:30:15,541 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower658] 12:30:15,542 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader655-notifier665#-131795094] 12:30:15,542 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader655-collector666#1879537632] 12:30:15,542 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader655] 12:30:15,543 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 12:30:15,543 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/15/2024 12:30:15.546] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower668]: org.opendaylight.controller.cluster.common.actor.Monitor@5d6eaf64 12:30:15,546 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower668 with journal batch size 1000 12:30:15,548 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668: SnapshotOffer called. 12:30:15,548 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668: Moving last applied index from -1 to -1 12:30:15,548 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668: Set currentTerm=1, votedFor=null 12:30:15,548 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668: Updated server config: isVoting: true, peers: [PeerInfo [id=leader667, address=akka://test/user/leader667, votingState=VOTING], PeerInfo [id=follower669, address=akka://test/user/follower669, votingState=VOTING], PeerInfo [id=follower670, address=akka://test/user/follower670, votingState=NON_VOTING]] 12:30:15,548 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower668 in 86.37 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [03/15/2024 12:30:15.549] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower669]: org.opendaylight.controller.cluster.common.actor.Monitor@5f104a0d 12:30:15,550 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower669 with journal batch size 1000 12:30:15,550 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669: SnapshotOffer called. 12:30:15,550 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669: Moving last applied index from -1 to -1 12:30:15,550 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669: Set currentTerm=1, votedFor=null 12:30:15,550 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669: Updated server config: isVoting: true, peers: [PeerInfo [id=leader667, address=akka://test/user/leader667, votingState=VOTING], PeerInfo [id=follower668, address=akka://test/user/follower668, votingState=VOTING], PeerInfo [id=follower670, address=akka://test/user/follower670, votingState=NON_VOTING]] 12:30:15,550 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower669 in 71.88 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:15,552 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower670 with journal batch size 1000 [WARN] [03/15/2024 12:30:15.552] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower670]: org.opendaylight.controller.cluster.common.actor.Monitor@3c0ed2aa 12:30:15,552 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower668: Recovery completed in 4.225 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:15,553 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower669: Recovery completed in 2.174 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:15,556 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: SnapshotOffer called. 12:30:15,556 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Moving last applied index from -1 to -1 12:30:15,556 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Set currentTerm=1, votedFor=null 12:30:15,556 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Updated server config: isVoting: false, peers: [PeerInfo [id=leader667, address=akka://test/user/leader667, votingState=VOTING], PeerInfo [id=follower669, address=akka://test/user/follower669, votingState=VOTING], PeerInfo [id=follower668, address=akka://test/user/follower668, votingState=VOTING]] 12:30:15,556 PM [test-akka.actor.default-dispatcher-10] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower670 in 117.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:15,557 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Recovery completed in 1.346 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/15/2024 12:30:15.655] [test-akka.actor.default-dispatcher-10] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader667]: org.opendaylight.controller.cluster.common.actor.Monitor@4fc8d4c1 12:30:15,656 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader667 with journal batch size 1000 12:30:15,657 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: SnapshotOffer called. 12:30:15,657 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: Moving last applied index from -1 to -1 12:30:15,657 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: Set currentTerm=1, votedFor=null 12:30:15,658 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: Updated server config: isVoting: true, peers: [PeerInfo [id=follower669, address=akka://test/user/follower669, votingState=VOTING], PeerInfo [id=follower668, address=akka://test/user/follower668, votingState=VOTING], PeerInfo [id=follower670, address=akka://test/user/follower670, votingState=NON_VOTING]] 12:30:15,658 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader667 in 109.4 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:15,659 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: Recovery completed in 1.643 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:16,3 PM [test-akka.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:30:16,4 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.170.99:25520, Removed) 12:30:16,4 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Follower): Received ElectionTimeout - switching to Candidate 12:30:16,4 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Candidate): Election: Candidate has following voting peers: [follower669, follower668] 12:30:16,4 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: Set currentTerm=2, votedFor=leader667 12:30:16,4 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Candidate): Starting new election term 2 12:30:16,4 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Candidate): Sending RequestVote [term=2, candidateId=leader667, lastLogIndex=-1, lastLogTerm=-1] to peer follower669 12:30:16,4 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Candidate): Sending RequestVote [term=2, candidateId=leader667, lastLogIndex=-1, lastLogTerm=-1] to peer follower668 12:30:16,4 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:16,4 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:30:16,4 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Follower): Cluster state: [] 12:30:16,4 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader667, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:16,4 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader667, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:16,4 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669: Set currentTerm=2, votedFor=null 12:30:16,4 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Follower): In requestVote: RequestVote [term=2, candidateId=leader667, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:16,4 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669: Set currentTerm=2, votedFor=leader667 12:30:16,4 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:16,5 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:16,5 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): Cluster state: [] 12:30:16,5 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader667, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:16,5 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader667, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:16,5 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668: Set currentTerm=2, votedFor=null 12:30:16,5 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): In requestVote: RequestVote [term=2, candidateId=leader667, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:16,5 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668: Set currentTerm=2, votedFor=leader667 12:30:16,5 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:16,6 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:30:16,6 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:30:16,7 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Election: Leader has following peers: [follower669, follower668, follower670] 12:30:16,7 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower669: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,7 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower668: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,7 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower670: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,7 PM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:30:16,7 PM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower669: Last sync leader does not match current leader leader667, need to catch up to -1 12:30:16,7 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower669 updated: matchIndex: -1, nextIndex: 0 12:30:16,7 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower669: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,7 PM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower670 (Follower): Term 2 in "AppendEntries [leaderId=leader667, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:30:16,7 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Set currentTerm=2, votedFor=null 12:30:16,8 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower668: Last sync leader does not match current leader leader667, need to catch up to -1 12:30:16,8 PM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower670: Last sync leader does not match current leader leader667, need to catch up to -1 [WARN] [03/15/2024 12:30:16.008] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower669] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@11ff452b 12:30:16,8 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower668 updated: matchIndex: -1, nextIndex: 0 12:30:16,8 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower668: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,8 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower670 updated: matchIndex: -1, nextIndex: 0 [WARN] [03/15/2024 12:30:16.008] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower668] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2650e36a 12:30:16,8 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower670: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/15/2024 12:30:16.013] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower670] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5755feab 12:30:16,14 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:30:16,14 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 12:30:16,17 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 12:30:16,17 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 12:30:16,17 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 12:30:16,18 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: onRequestLeadership RequestLeadership [requestedFollowerId=follower669, replyTo=Actor[akka://test/user/$a#1765210882]] 12:30:16,19 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: Initiating leader transfer 12:30:16,19 PM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader667: pauseLeader successfully completed - doing transfer 12:30:16,19 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Attempting to transfer leadership 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669: onLeaderTransitioning: LeaderTransitioning [leaderId=leader667] 12:30:16,20 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower669: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668: onLeaderTransitioning: LeaderTransitioning [leaderId=leader667] 12:30:16,20 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower668: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: onLeaderTransitioning: LeaderTransitioning [leaderId=leader667] 12:30:16,20 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower670: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower669: Lagging 0 entries behind leader leader667 and reached -1 (of expected -1) 12:30:16,20 PM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower668: Lagging 0 entries behind leader leader667 and reached -1 (of expected -1) 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): tryToCompleteLeadershipTransfer: followerId: follower669, matchIndex: -1, lastIndex: -1, isVoting: true 12:30:16,20 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower670: Lagging 0 entries behind leader leader667 and reached -1 (of expected -1) 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Follower's log matches - sending ElectionTimeout 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower669: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/15/2024 12:30:16.020] [test-akka.actor.default-dispatcher-10] [akka://test/user/follower669] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3d6560f 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower668: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Checking sendAppendEntries for follower follower670: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Leader transfer complete 12:30:16,20 PM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader667: leader transfer complete - waiting for new leader 12:30:16,20 PM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower669: Lagging 0 entries behind leader leader667 and reached -1 (of expected -1) [WARN] [03/15/2024 12:30:16.020] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower670] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@581e3f7a 12:30:16,20 PM [test-akka.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:30:16,20 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Follower): Received TimeoutNow - switching to Candidate 12:30:16,20 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower670: Lagging 0 entries behind leader leader667 and reached -1 (of expected -1) 12:30:16,20 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Candidate): Election: Candidate has following voting peers: [leader667, follower668] 12:30:16,20 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669: Set currentTerm=3, votedFor=follower669 12:30:16,21 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Candidate): Starting new election term 3 12:30:16,21 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Candidate): Sending RequestVote [term=3, candidateId=follower669, lastLogIndex=-1, lastLogTerm=-1] to peer leader667 12:30:16,21 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Candidate): Sending RequestVote [term=3, candidateId=follower669, lastLogIndex=-1, lastLogTerm=-1] to peer follower668 12:30:16,21 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:16,21 PM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower668: Lagging 0 entries behind leader leader667 and reached -1 (of expected -1) 12:30:16,21 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Cluster state: [] 12:30:16,21 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower669, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. [WARN] [03/15/2024 12:30:16.021] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower668] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@11464e7c 12:30:16,21 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower669, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 12:30:16,21 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: Set currentTerm=3, votedFor=null 12:30:16,21 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 12:30:16,21 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): Leadership transfer in progress - processing RequestVote 12:30:16,21 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:16,21 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): Cluster state: [] 12:30:16,21 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower669, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:16,21 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower669, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 2 - updating term 12:30:16,21 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668: Set currentTerm=3, votedFor=null 12:30:16,22 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): In requestVote: RequestVote [term=3, candidateId=follower669, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:16,22 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668: Set currentTerm=3, votedFor=follower669 12:30:16,22 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower668 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 12:30:16,22 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): In requestVote: RequestVote [term=3, candidateId=follower669, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:16,22 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: Set currentTerm=3, votedFor=follower669 12:30:16,22 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 12:30:16,22 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader667 (Leader) :- Switching from behavior Leader to Follower, election term: 3 12:30:16,24 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 12:30:16,24 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 12:30:16,25 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Leader): Election: Leader has following peers: [leader667, follower668, follower670] 12:30:16,25 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Leader): Checking sendAppendEntries for follower leader667: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,25 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Leader): Checking sendAppendEntries for follower follower668: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,25 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Leader): Checking sendAppendEntries for follower follower670: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,25 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 12:30:16,25 PM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower668: Last sync leader does not match current leader follower669, need to catch up to -1 12:30:16,26 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower670 (Follower): Term 3 in "AppendEntries [leaderId=follower669, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 12:30:16,26 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Set currentTerm=3, votedFor=null 12:30:16,26 PM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower670: Last sync leader does not match current leader follower669, need to catch up to -1 12:30:16,26 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower668 updated: matchIndex: -1, nextIndex: 0 12:30:16,26 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Leader): Checking sendAppendEntries for follower follower668: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,26 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower670 updated: matchIndex: -1, nextIndex: 0 12:30:16,26 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Leader): Checking sendAppendEntries for follower follower670: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,26 PM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - leader667: Last sync leader does not match current leader follower669, need to catch up to -1 12:30:16,26 PM [test-akka.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader667: leader changed to follower669 [WARN] [03/15/2024 12:30:16.026] [test-akka.actor.default-dispatcher-10] [akka://test/user/follower668] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@372ee02f 12:30:16,26 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader667 updated: matchIndex: -1, nextIndex: 0 12:30:16,27 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower669 (Leader): Checking sendAppendEntries for follower leader667: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,27 PM [test-akka.actor.default-dispatcher-6] [INFO] RaftActorLeadershipTransferCohort - leader667: Successfully transferred leadership to follower669 in 6.961 ms 12:30:16,27 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader667: Leadership transferred successfully to follower669 [WARN] [03/15/2024 12:30:16.027] [test-akka.actor.default-dispatcher-16] [akka://test/user/leader667] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7d165f08 [WARN] [03/15/2024 12:30:16.028] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower670] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@50f9e311 12:30:16,69 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 12:30:16,70 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower668-notifier671#-1498325056] 12:30:16,81 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower668-collector672#1994230792] 12:30:16,81 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower668] 12:30:16,81 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower669-notifier673#-1794511716] 12:30:16,82 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower669-collector674#996958011] 12:30:16,82 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower669] 12:30:16,82 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower670-notifier675#81704073] 12:30:16,83 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower670-collector676#1344032040] 12:30:16,83 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower670] 12:30:16,83 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader667-notifier677#1858317908] 12:30:16,83 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader667-collector678#-1262434918] 12:30:16,83 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader667] 12:30:16,84 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$a#1765210882] 12:30:16,85 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 12:30:16,85 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/15/2024 12:30:16.088] [test-akka.actor.default-dispatcher-16] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower680]: org.opendaylight.controller.cluster.common.actor.Monitor@13d89be1 12:30:16,89 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower680 with journal batch size 1000 12:30:16,90 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680: SnapshotOffer called. 12:30:16,90 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680: Moving last applied index from -1 to -1 12:30:16,90 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680: Set currentTerm=1, votedFor=null 12:30:16,90 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680: Updated server config: isVoting: true, peers: [PeerInfo [id=leader679, address=akka://test/user/leader679, votingState=VOTING], PeerInfo [id=follower681, address=akka://test/user/follower681, votingState=VOTING], PeerInfo [id=follower682, address=akka://test/user/follower682, votingState=NON_VOTING]] 12:30:16,90 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower680 in 99.77 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:16,91 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower680: Recovery completed in 1.498 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/15/2024 12:30:16.091] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower681]: org.opendaylight.controller.cluster.common.actor.Monitor@760b216 12:30:16,92 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower681 with journal batch size 1000 12:30:16,92 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681: SnapshotOffer called. 12:30:16,92 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681: Moving last applied index from -1 to -1 12:30:16,92 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681: Set currentTerm=1, votedFor=null 12:30:16,92 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681: Updated server config: isVoting: true, peers: [PeerInfo [id=leader679, address=akka://test/user/leader679, votingState=VOTING], PeerInfo [id=follower680, address=akka://test/user/follower680, votingState=VOTING], PeerInfo [id=follower682, address=akka://test/user/follower682, votingState=NON_VOTING]] 12:30:16,93 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower681 in 80.51 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [03/15/2024 12:30:16.094] [test-akka.actor.default-dispatcher-16] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower682]: org.opendaylight.controller.cluster.common.actor.Monitor@5cf9688d 12:30:16,95 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower681: Recovery completed in 1.712 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:16,95 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower682 with journal batch size 1000 [WARN] [03/15/2024 12:30:16.097] [test-akka.actor.default-dispatcher-10] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader679]: org.opendaylight.controller.cluster.common.actor.Monitor@1c1ced19 12:30:16,97 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader679 with journal batch size 1000 12:30:16,101 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679: SnapshotOffer called. 12:30:16,101 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679: Moving last applied index from -1 to -1 12:30:16,101 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679: Set currentTerm=1, votedFor=null 12:30:16,101 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: SnapshotOffer called. 12:30:16,101 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679: Updated server config: isVoting: true, peers: [PeerInfo [id=follower681, address=akka://test/user/follower681, votingState=VOTING], PeerInfo [id=follower680, address=akka://test/user/follower680, votingState=VOTING], PeerInfo [id=follower682, address=akka://test/user/follower682, votingState=NON_VOTING]] 12:30:16,101 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Moving last applied index from -1 to -1 12:30:16,101 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Set currentTerm=1, votedFor=null 12:30:16,101 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Updated server config: isVoting: false, peers: [PeerInfo [id=leader679, address=akka://test/user/leader679, votingState=VOTING], PeerInfo [id=follower681, address=akka://test/user/follower681, votingState=VOTING], PeerInfo [id=follower680, address=akka://test/user/follower680, votingState=VOTING]] 12:30:16,101 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader679 in 101.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:16,102 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower682 in 119.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:16,102 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader679: Recovery completed in 937.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:16,103 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Recovery completed in 1.040 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:16,493 PM [test-akka.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:30:16,493 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.170.99:25520, Removed) 12:30:16,493 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Follower): Received ElectionTimeout - switching to Candidate 12:30:16,493 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Candidate): Election: Candidate has following voting peers: [follower681, follower680] 12:30:16,493 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679: Set currentTerm=2, votedFor=leader679 12:30:16,494 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Candidate): Starting new election term 2 12:30:16,494 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Candidate): Sending RequestVote [term=2, candidateId=leader679, lastLogIndex=-1, lastLogTerm=-1] to peer follower681 12:30:16,494 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Candidate): Sending RequestVote [term=2, candidateId=leader679, lastLogIndex=-1, lastLogTerm=-1] to peer follower680 12:30:16,494 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:30:16,494 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:16,494 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681 (Follower): Cluster state: [] 12:30:16,494 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader679, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:16,494 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower681 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader679, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:16,494 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681: Set currentTerm=2, votedFor=null 12:30:16,494 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:16,494 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680 (Follower): Cluster state: [] 12:30:16,494 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader679, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:16,494 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower680 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader679, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:16,494 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680: Set currentTerm=2, votedFor=null 12:30:16,495 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680 (Follower): In requestVote: RequestVote [term=2, candidateId=leader679, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:16,495 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680: Set currentTerm=2, votedFor=leader679 12:30:16,495 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:16,496 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681 (Follower): In requestVote: RequestVote [term=2, candidateId=leader679, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:16,496 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681: Set currentTerm=2, votedFor=leader679 12:30:16,496 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:16,496 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:30:16,496 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:30:16,496 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Election: Leader has following peers: [follower681, follower680, follower682] 12:30:16,496 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Checking sendAppendEntries for follower follower681: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,496 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Checking sendAppendEntries for follower follower680: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,496 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Checking sendAppendEntries for follower follower682: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,496 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:30:16,497 PM [test-akka.actor.default-dispatcher-10] [DEBUG] SyncStatusTracker - follower681: Last sync leader does not match current leader leader679, need to catch up to -1 12:30:16,497 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower681 updated: matchIndex: -1, nextIndex: 0 12:30:16,497 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Checking sendAppendEntries for follower follower681: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,497 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Term 2 in "AppendEntries [leaderId=leader679, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:30:16,497 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Set currentTerm=2, votedFor=null 12:30:16,497 PM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower680: Last sync leader does not match current leader leader679, need to catch up to -1 12:30:16,497 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower682: Last sync leader does not match current leader leader679, need to catch up to -1 12:30:16,497 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower680 updated: matchIndex: -1, nextIndex: 0 [WARN] [03/15/2024 12:30:16.497] [test-akka.actor.default-dispatcher-10] [akka://test/user/follower681] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@15fb4c87 12:30:16,497 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Checking sendAppendEntries for follower follower680: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,498 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower682 updated: matchIndex: -1, nextIndex: 0 [WARN] [03/15/2024 12:30:16.497] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower680] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3aacf6cd 12:30:16,498 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Checking sendAppendEntries for follower follower682: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/15/2024 12:30:16.498] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower682] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4282eecb 12:30:16,508 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:30:16,508 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 12:30:16,509 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 12:30:16,509 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower681 starting 12:30:16,509 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower681: onShutDown 12:30:16,510 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower681 ending 12:30:16,510 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 12:30:16,510 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 12:30:16,510 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679: onRequestLeadership RequestLeadership [requestedFollowerId=follower681, replyTo=Actor[akka://test/user/$b#-575028695]] 12:30:16,510 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679: Initiating leader transfer 12:30:16,510 PM [test-akka.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader679: pauseLeader successfully completed - doing transfer 12:30:16,510 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Attempting to transfer leadership 12:30:16,510 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Checking sendAppendEntries for follower follower681: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,510 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Checking sendAppendEntries for follower follower680: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:16,510 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Checking sendAppendEntries for follower follower682: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/15/2024 12:30:16.510] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower681] received dead letter from TestActor[akka://test/user/leader679]: LeaderTransitioning [leaderId=leader679] [WARN] [03/15/2024 12:30:16.511] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower681] received dead letter from TestActor[akka://test/user/leader679]: AppendEntries [leaderId=leader679, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:30:16,511 PM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower682: Lagging 0 entries behind leader leader679 and reached -1 (of expected -1) 12:30:16,511 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: onLeaderTransitioning: LeaderTransitioning [leaderId=leader679] [WARN] [03/15/2024 12:30:16.511] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower682] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@27fc5b36 12:30:16,511 PM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower680: Lagging 0 entries behind leader leader679 and reached -1 (of expected -1) 12:30:16,511 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower680: onLeaderTransitioning: LeaderTransitioning [leaderId=leader679] [WARN] [03/15/2024 12:30:16.512] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower680] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@309304b [WARN] [03/15/2024 12:30:16.613] [test-akka.actor.default-dispatcher-10] [akka://test/user/follower681] received dead letter from TestActor[akka://test/user/leader679]: AppendEntries [leaderId=leader679, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:30:16,613 PM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower682: Lagging 0 entries behind leader leader679 and reached -1 (of expected -1) 12:30:16,613 PM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower680: Lagging 0 entries behind leader leader679 and reached -1 (of expected -1) [WARN] [03/15/2024 12:30:16.734] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower681] received dead letter from TestActor[akka://test/user/leader679]: AppendEntries [leaderId=leader679, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:30:16,734 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower680: Lagging 0 entries behind leader leader679 and reached -1 (of expected -1) 12:30:16,734 PM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower682: Lagging 0 entries behind leader leader679 and reached -1 (of expected -1) 12:30:16,853 PM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader679: leader transfer aborted 12:30:16,853 PM [test-akka.actor.default-dispatcher-17] [WARN] RaftActorLeadershipTransferCohort - leader679: Failed to transfer leadership in 342.9 ms 12:30:16,853 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679: LeadershipTransfer request from follower681 failed 12:30:16,854 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader679 (Leader): Leadership transfer expired [WARN] [03/15/2024 12:30:16.855] [test-akka.actor.default-dispatcher-10] [akka://test/user/follower681] received dead letter from TestActor[akka://test/user/leader679]: AppendEntries [leaderId=leader679, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:30:16,855 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower680: Lagging 0 entries behind leader leader679 and reached -1 (of expected -1) 12:30:16,855 PM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower682: Lagging 0 entries behind leader leader679 and reached -1 (of expected -1) 12:30:16,865 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 12:30:16,865 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower680-notifier683#1978253277] 12:30:16,876 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower680-collector684#-2125636870] 12:30:16,876 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower680] 12:30:16,879 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower681-notifier685#-1747123707] 12:30:16,879 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower681-collector686#1124576060] 12:30:16,880 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower681] 12:30:16,880 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower682-notifier687#-1739989990] 12:30:16,880 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower682-collector688#902253209] 12:30:16,880 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower682] 12:30:16,881 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader679-notifier689#-1305122600] 12:30:16,881 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader679-collector690#947558441] 12:30:16,882 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader679] 12:30:16,882 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$b#-575028695] 12:30:16,885 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 12:30:16,885 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/15/2024 12:30:16.887] [test-akka.actor.default-dispatcher-10] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower692]: org.opendaylight.controller.cluster.common.actor.Monitor@1f4ce89f 12:30:16,888 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower692 with journal batch size 1000 12:30:16,889 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: SnapshotOffer called. 12:30:16,889 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Moving last applied index from -1 to -1 12:30:16,889 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Set currentTerm=1, votedFor=null 12:30:16,889 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Updated server config: isVoting: true, peers: [PeerInfo [id=follower694, address=akka://test/user/follower694, votingState=NON_VOTING], PeerInfo [id=follower693, address=akka://test/user/follower693, votingState=VOTING], PeerInfo [id=leader691, address=akka://test/user/leader691, votingState=VOTING]] 12:30:16,889 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower692 in 74.11 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:16,890 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Recovery completed in 1.066 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/15/2024 12:30:16.990] [test-akka.actor.default-dispatcher-15] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower693]: org.opendaylight.controller.cluster.common.actor.Monitor@3b437915 12:30:16,991 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower693 with journal batch size 1000 12:30:16,991 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: SnapshotOffer called. 12:30:16,991 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Moving last applied index from -1 to -1 12:30:16,991 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Set currentTerm=1, votedFor=null 12:30:16,992 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Updated server config: isVoting: true, peers: [PeerInfo [id=follower692, address=akka://test/user/follower692, votingState=VOTING], PeerInfo [id=follower694, address=akka://test/user/follower694, votingState=NON_VOTING], PeerInfo [id=leader691, address=akka://test/user/leader691, votingState=VOTING]] 12:30:16,992 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower693 in 105.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:16,993 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Recovery completed in 1.722 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/15/2024 12:30:16.994] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower694]: org.opendaylight.controller.cluster.common.actor.Monitor@5a48b5d9 12:30:16,995 PM [test-akka.actor.default-dispatcher-10] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower694 with journal batch size 1000 12:30:16,999 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: SnapshotOffer called. 12:30:16,999 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: Moving last applied index from -1 to -1 12:30:16,999 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: Set currentTerm=1, votedFor=null 12:30:16,999 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: Updated server config: isVoting: false, peers: [PeerInfo [id=follower692, address=akka://test/user/follower692, votingState=VOTING], PeerInfo [id=follower693, address=akka://test/user/follower693, votingState=VOTING], PeerInfo [id=leader691, address=akka://test/user/leader691, votingState=VOTING]] 12:30:17,0 PM [test-akka.actor.default-dispatcher-10] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower694 in 242.4 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [03/15/2024 12:30:17.000] [test-akka.actor.default-dispatcher-15] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader691]: org.opendaylight.controller.cluster.common.actor.Monitor@844320f 12:30:17,1 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader691 with journal batch size 1000 12:30:17,1 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: Recovery completed in 2.277 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:17,2 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: SnapshotOffer called. 12:30:17,2 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Moving last applied index from -1 to -1 12:30:17,2 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Set currentTerm=1, votedFor=null 12:30:17,2 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Updated server config: isVoting: true, peers: [PeerInfo [id=follower692, address=akka://test/user/follower692, votingState=VOTING], PeerInfo [id=follower694, address=akka://test/user/follower694, votingState=NON_VOTING], PeerInfo [id=follower693, address=akka://test/user/follower693, votingState=VOTING]] 12:30:17,3 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader691 in 112.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:17,3 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Recovery completed in 878.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:17,374 PM [test-akka.actor.default-dispatcher-10] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:30:17,374 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.170.99:25520, Removed) 12:30:17,374 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Follower): Received ElectionTimeout - switching to Candidate 12:30:17,374 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Candidate): Election: Candidate has following voting peers: [follower692, follower693] 12:30:17,374 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Set currentTerm=2, votedFor=leader691 12:30:17,374 PM [test-akka.actor.default-dispatcher-10] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Candidate): Starting new election term 2 12:30:17,374 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Candidate): Sending RequestVote [term=2, candidateId=leader691, lastLogIndex=-1, lastLogTerm=-1] to peer follower692 12:30:17,374 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Candidate): Sending RequestVote [term=2, candidateId=leader691, lastLogIndex=-1, lastLogTerm=-1] to peer follower693 12:30:17,374 PM [test-akka.actor.default-dispatcher-10] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:30:17,374 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:17,375 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): Cluster state: [] 12:30:17,375 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:17,375 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader691, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:17,375 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader691, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:17,375 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Set currentTerm=2, votedFor=null 12:30:17,375 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Cluster state: [] 12:30:17,375 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader691, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:17,375 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader691, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:17,375 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Set currentTerm=2, votedFor=null 12:30:17,375 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): In requestVote: RequestVote [term=2, candidateId=leader691, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:17,375 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Set currentTerm=2, votedFor=leader691 12:30:17,375 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:17,375 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): In requestVote: RequestVote [term=2, candidateId=leader691, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:17,375 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Set currentTerm=2, votedFor=leader691 12:30:17,375 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:17,376 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:30:17,376 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:30:17,376 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Election: Leader has following peers: [follower692, follower694, follower693] 12:30:17,376 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower692: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:17,376 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower694: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:17,376 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower693: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:17,376 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower692: Last sync leader does not match current leader leader691, need to catch up to -1 12:30:17,376 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:30:17,376 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): Term 2 in "AppendEntries [leaderId=leader691, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:30:17,377 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: Set currentTerm=2, votedFor=null 12:30:17,377 PM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower693: Last sync leader does not match current leader leader691, need to catch up to -1 12:30:17,377 PM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower694: Last sync leader does not match current leader leader691, need to catch up to -1 12:30:17,377 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower692 updated: matchIndex: -1, nextIndex: 0 12:30:17,377 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower692: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:17,377 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower694 updated: matchIndex: -1, nextIndex: 0 [WARN] [03/15/2024 12:30:17.377] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower692] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7ab50f82 12:30:17,377 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower694: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/15/2024 12:30:17.377] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower693] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@eb7fd0d 12:30:17,377 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower693 updated: matchIndex: -1, nextIndex: 0 12:30:17,377 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower693: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/15/2024 12:30:17.378] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower694] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@180300c8 12:30:17,408 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:30:17,408 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 12:30:17,408 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:17,408 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:17,409 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 12:30:17,409 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower692: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,409 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): sendAppendEntries: 0 is present for follower follower692 12:30:17,409 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Sending AppendEntries to follower follower692: AppendEntries [leaderId=leader691, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:17,409 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower694: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,409 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): sendAppendEntries: 0 is present for follower follower694 12:30:17,409 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Sending AppendEntries to follower follower694: AppendEntries [leaderId=leader691, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:17,409 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower693: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,409 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): sendAppendEntries: 0 is present for follower follower693 12:30:17,409 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Sending AppendEntries to follower follower693: AppendEntries [leaderId=leader691, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:17,409 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): handleAppendEntries: AppendEntries [leaderId=leader691, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:17,409 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): Number of entries to be appended = 1 12:30:17,409 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 12:30:17,409 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): Append entry to log zero 12:30:17,409 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:17,409 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): handleAppendEntries: AppendEntries [leaderId=leader691, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:17,409 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): Number of entries to be appended = 1 12:30:17,409 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 12:30:17,409 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): Append entry to log zero 12:30:17,409 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:17,410 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): Log size is now 1 12:30:17,410 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): Log size is now 1 12:30:17,410 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower694, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:30:17,410 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower692, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:30:17,410 PM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower694: Lagging 0 entries behind leader leader691 and reached -1 (of expected -1) 12:30:17,410 PM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower692: Lagging 0 entries behind leader leader691 and reached -1 (of expected -1) 12:30:17,410 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower694 updated: matchIndex: 0, nextIndex: 1 12:30:17,410 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower694: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,410 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower692 updated: matchIndex: 0, nextIndex: 1 12:30:17,410 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower692: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,410 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:17,411 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 12:30:17,411 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Setting last applied to 0 12:30:17,411 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Moving last applied index from -1 to 0 12:30:17,411 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Applying state for log index 0 data zero 12:30:17,411 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: applyState called: zero 12:30:17,411 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Persisting ApplyJournalEntries with index=0 12:30:17,411 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:17,411 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [03/15/2024 12:30:17.411] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower694] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3c3d5717 [WARN] [03/15/2024 12:30:17.412] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower692] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6fcf0032 12:30:17,493 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): sendAppendEntries: 0 is present for follower follower693 12:30:17,493 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Sending AppendEntries to follower follower693: AppendEntries [leaderId=leader691, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:17,613 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): sendAppendEntries: 0 is present for follower follower693 12:30:17,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Sending AppendEntries to follower follower693: AppendEntries [leaderId=leader691, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:17,614 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): Commit index set to 0 12:30:17,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): Commit index set to 0 12:30:17,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] SyncStatusTracker - follower694: Lagging 0 entries behind leader leader691 and reached 0 (of expected -1) 12:30:17,614 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower692: Lagging 0 entries behind leader leader691 and reached 0 (of expected -1) 12:30:17,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 12:30:17,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): Setting last applied to 0 12:30:17,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: Moving last applied index from -1 to 0 12:30:17,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: Applying state for log index 0 data zero 12:30:17,614 PM [test-akka.actor.default-dispatcher-10] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: applyState called: zero 12:30:17,614 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 12:30:17,614 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: Persisting ApplyJournalEntries with index=0 12:30:17,614 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): Setting last applied to 0 12:30:17,614 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Moving last applied index from -1 to 0 12:30:17,614 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Applying state for log index 0 data zero 12:30:17,615 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: applyState called: zero 12:30:17,615 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Persisting ApplyJournalEntries with index=0 12:30:17,661 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 12:30:17,661 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 12:30:17,661 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: onShutDown 12:30:17,661 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Initiating leader transfer 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader691: pauseLeader successfully completed - doing transfer 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Attempting to transfer leadership 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower692: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower694: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower693: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): sendAppendEntries: 0 is present for follower follower693 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower694: Lagging 0 entries behind leader leader691 and reached 0 (of expected -1) 12:30:17,662 PM [test-akka.actor.default-dispatcher-10] [DEBUG] SyncStatusTracker - follower692: Lagging 0 entries behind leader leader691 and reached 0 (of expected -1) 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: onLeaderTransitioning: LeaderTransitioning [leaderId=leader691] 12:30:17,662 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: onLeaderTransitioning: LeaderTransitioning [leaderId=leader691] 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): tryToCompleteLeadershipTransfer: followerId: follower694, matchIndex: 0, lastIndex: 0, isVoting: false 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): tryToCompleteLeadershipTransfer: followerId: follower692, matchIndex: 0, lastIndex: 0, isVoting: true 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Follower's log matches - sending ElectionTimeout 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower692: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower694: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Checking sendAppendEntries for follower follower693: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): sendAppendEntries: 0 is present for follower follower693 12:30:17,662 PM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower692: Lagging 0 entries behind leader leader691 and reached 0 (of expected -1) 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Leader transfer complete 12:30:17,662 PM [test-akka.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader691: leader transfer complete - waiting for new leader 12:30:17,662 PM [test-akka.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:30:17,662 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower): Received TimeoutNow - switching to Candidate 12:30:17,662 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower694: Lagging 0 entries behind leader leader691 and reached 0 (of expected -1) 12:30:17,662 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Candidate): Election: Candidate has following voting peers: [follower693, leader691] 12:30:17,662 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692: Set currentTerm=3, votedFor=follower692 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Candidate): Starting new election term 3 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Candidate): Sending RequestVote [term=3, candidateId=follower692, lastLogIndex=0, lastLogTerm=2] to peer follower693 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Candidate): Sending RequestVote [term=3, candidateId=follower692, lastLogIndex=0, lastLogTerm=2] to peer leader691 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Cluster state: [] 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower692, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower692, lastLogIndex=0, lastLogTerm=2]" message is greater than follower's term 2 - updating term 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Set currentTerm=3, votedFor=null 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): In requestVote: RequestVote [term=3, candidateId=follower692, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Set currentTerm=3, votedFor=follower692 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Cluster state: [] 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower692, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower692, lastLogIndex=0, lastLogTerm=2]" message is greater than leader's term 2 - switching to Follower 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Set currentTerm=3, votedFor=null 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): Leadership transfer in progress - processing RequestVote 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): In requestVote: RequestVote [term=3, candidateId=follower692, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: Set currentTerm=3, votedFor=follower692 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 12:30:17,663 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader691 (Leader) :- Switching from behavior Leader to Follower, election term: 3 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): Election: Leader has following peers: [follower694, follower693, leader691] 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): Checking sendAppendEntries for follower follower694: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): Checking sendAppendEntries for follower follower693: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): Checking sendAppendEntries for follower leader691: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 12:30:17,664 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower694 (Follower): Term 3 in "AppendEntries [leaderId=follower692, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 12:30:17,664 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower694: Set currentTerm=3, votedFor=null 12:30:17,664 PM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower694: Last sync leader does not match current leader follower692, need to catch up to -1 12:30:17,664 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - leader691: Last sync leader does not match current leader follower692, need to catch up to -1 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower694 updated: matchIndex: 0, nextIndex: 1 12:30:17,664 PM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader691: leader changed to follower692 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): Checking sendAppendEntries for follower follower694: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader691 updated: matchIndex: 0, nextIndex: 1 12:30:17,664 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): Checking sendAppendEntries for follower leader691: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:17,665 PM [test-akka.actor.default-dispatcher-17] [INFO] RaftActorLeadershipTransferCohort - leader691: Successfully transferred leadership to follower692 in 3.043 ms 12:30:17,665 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader691: leader transfer succeeded - sending PoisonPill 12:30:17,665 PM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - leader691: Lagging 0 entries behind leader follower692 and reached 0 (of expected -1) [WARN] [03/15/2024 12:30:17.665] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader691] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@56754907 [WARN] [03/15/2024 12:30:17.665] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader691] received dead letter: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@ad37734 [WARN] [03/15/2024 12:30:17.665] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower694] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3b373b95 12:30:17,665 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower694: Lagging 0 entries behind leader follower692 and reached 0 (of expected -1) [WARN] [03/15/2024 12:30:17.665] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower694] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@758fe0fc 12:30:18,184 PM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower694: Lagging 0 entries behind leader follower692 and reached 0 (of expected -1) 12:30:18,184 PM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower693: Last sync leader does not match current leader follower692, need to catch up to -1 12:30:18,184 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): sendAppendEntries: 0 is present for follower follower693 [WARN] [03/15/2024 12:30:18.184] [test-akka.actor.default-dispatcher-6] [akka://test/user/leader691] received dead letter from TestActor[akka://test/user/follower692]: AppendEntries [leaderId=follower692, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [03/15/2024 12:30:18.184] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower693] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@40622c3f 12:30:18,185 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): Sending AppendEntries to follower follower693: AppendEntries [leaderId=follower692, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:18,185 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): handleAppendEntries: AppendEntries [leaderId=follower692, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:18,185 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Number of entries to be appended = 1 12:30:18,185 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 12:30:18,185 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Append entry to log zero 12:30:18,185 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:18,185 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Log size is now 1 12:30:18,185 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Commit index set to 0 12:30:18,186 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): handleAppendEntries returning : AppendEntriesReply [term=3, success=true, followerId=follower693, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:30:18,186 PM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower693: Lagging 0 entries behind leader follower692 and reached 0 (of expected -1) 12:30:18,186 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 12:30:18,186 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693 (Follower): Setting last applied to 0 12:30:18,186 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Moving last applied index from -1 to 0 12:30:18,186 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Applying state for log index 0 data zero 12:30:18,186 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower693 updated: matchIndex: 0, nextIndex: 1 12:30:18,186 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower692 (Leader): Checking sendAppendEntries for follower follower693: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:18,186 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: applyState called: zero 12:30:18,187 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:18,187 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: Persisting ApplyJournalEntries with index=0 [WARN] [03/15/2024 12:30:18.188] [test-akka.actor.default-dispatcher-10] [akka://test/user/follower693] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@60635eac 12:30:18,218 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: onLeaderTransitioning: LeaderTransitioning [leaderId=leader691] 12:30:18,219 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 12:30:18,219 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower693 starting 12:30:18,219 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower693: onShutDown 12:30:18,220 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower693 ending 12:30:18,220 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 12:30:18,220 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower692-notifier695#701363861] 12:30:18,232 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower692-collector696#-268016750] 12:30:18,232 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower692] 12:30:18,232 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower693-notifier697#2123828120] 12:30:18,233 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower693-collector698#1569927598] 12:30:18,233 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower693] 12:30:18,233 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower694-notifier699#1591387417] 12:30:18,233 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower694-collector700#-1874782588] 12:30:18,233 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower694] 12:30:18,233 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader691-notifier701#-1257218025] 12:30:18,233 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader691-collector702#-1139646206] 12:30:18,233 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader691] 12:30:18,234 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [03/15/2024 12:30:18.236] [test-akka.actor.default-dispatcher-16] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader703]: org.opendaylight.controller.cluster.common.actor.Monitor@468685fa 12:30:18,237 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader703 with journal batch size 1000 12:30:18,238 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader703: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:18,239 PM [test-akka.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:30:18,239 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader703 (Follower): Received TimeoutNow - switching to Candidate 12:30:18,239 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader703 (Candidate): Election: Candidate has following voting peers: [] 12:30:18,239 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader703: Set currentTerm=1, votedFor=leader703 12:30:18,239 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader703 (Candidate): Starting new election term 1 12:30:18,239 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader703 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 12:30:18,240 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader703 (Candidate): Received ElectionTimeout 12:30:18,240 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader703 (Leader): Election: Leader has following peers: [] 12:30:18,240 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader703 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 12:30:18,241 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader703 starting 12:30:18,241 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader703: onShutDown 12:30:18,242 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader703 ending 12:30:18,242 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 12:30:18,242 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader703-collector707#-1680427537] 12:30:18,253 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader703] 12:30:18,255 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 12:30:18,255 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/15/2024 12:30:18.257] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower709]: org.opendaylight.controller.cluster.common.actor.Monitor@78d4dcc7 12:30:18,258 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower709 with journal batch size 1000 12:30:18,259 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: SnapshotOffer called. 12:30:18,259 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: Moving last applied index from -1 to -1 12:30:18,259 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: Set currentTerm=1, votedFor=null 12:30:18,259 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: Updated server config: isVoting: true, peers: [PeerInfo [id=follower711, address=akka://test/user/follower711, votingState=NON_VOTING], PeerInfo [id=follower710, address=akka://test/user/follower710, votingState=VOTING], PeerInfo [id=leader708, address=akka://test/user/leader708, votingState=VOTING]] 12:30:18,259 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower709 in 112.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [03/15/2024 12:30:18.260] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower710]: org.opendaylight.controller.cluster.common.actor.Monitor@5007b88b 12:30:18,260 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower710 with journal batch size 1000 12:30:18,261 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710: SnapshotOffer called. 12:30:18,261 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: Recovery completed in 2.278 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:18,261 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710: Moving last applied index from -1 to -1 12:30:18,261 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710: Set currentTerm=1, votedFor=null 12:30:18,261 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710: Updated server config: isVoting: true, peers: [PeerInfo [id=follower711, address=akka://test/user/follower711, votingState=NON_VOTING], PeerInfo [id=leader708, address=akka://test/user/leader708, votingState=VOTING], PeerInfo [id=follower709, address=akka://test/user/follower709, votingState=VOTING]] 12:30:18,262 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower710 in 155.4 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [03/15/2024 12:30:18.262] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower711]: org.opendaylight.controller.cluster.common.actor.Monitor@7cd0048b 12:30:18,263 PM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower711 with journal batch size 1000 12:30:18,264 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: SnapshotOffer called. 12:30:18,264 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Moving last applied index from -1 to -1 12:30:18,264 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Set currentTerm=1, votedFor=null 12:30:18,264 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Updated server config: isVoting: false, peers: [PeerInfo [id=follower710, address=akka://test/user/follower710, votingState=VOTING], PeerInfo [id=leader708, address=akka://test/user/leader708, votingState=VOTING], PeerInfo [id=follower709, address=akka://test/user/follower709, votingState=VOTING]] 12:30:18,264 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower711 in 56.56 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:18,264 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower710: Recovery completed in 2.701 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:18,265 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Recovery completed in 1.113 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/15/2024 12:30:18.365] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader708]: org.opendaylight.controller.cluster.common.actor.Monitor@4a101d3d 12:30:18,365 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader708 with journal batch size 1000 12:30:18,366 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: SnapshotOffer called. 12:30:18,366 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Moving last applied index from -1 to -1 12:30:18,366 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Set currentTerm=1, votedFor=null 12:30:18,366 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Updated server config: isVoting: true, peers: [PeerInfo [id=follower711, address=akka://test/user/follower711, votingState=NON_VOTING], PeerInfo [id=follower710, address=akka://test/user/follower710, votingState=VOTING], PeerInfo [id=follower709, address=akka://test/user/follower709, votingState=VOTING]] 12:30:18,367 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader708 in 84.52 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:18,369 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Recovery completed in 1.970 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:18,773 PM [test-akka.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:30:18,773 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.170.99:25520, Removed) 12:30:18,773 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Follower): Received ElectionTimeout - switching to Candidate 12:30:18,774 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Candidate): Election: Candidate has following voting peers: [follower710, follower709] 12:30:18,774 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Set currentTerm=2, votedFor=leader708 12:30:18,774 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Candidate): Starting new election term 2 12:30:18,774 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Candidate): Sending RequestVote [term=2, candidateId=leader708, lastLogIndex=-1, lastLogTerm=-1] to peer follower710 12:30:18,774 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Candidate): Sending RequestVote [term=2, candidateId=leader708, lastLogIndex=-1, lastLogTerm=-1] to peer follower709 12:30:18,774 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:30:18,774 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:18,774 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710 (Follower): Cluster state: [] 12:30:18,774 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader708, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:18,774 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower710 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader708, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:18,774 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710: Set currentTerm=2, votedFor=null 12:30:18,775 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:18,775 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710 (Follower): In requestVote: RequestVote [term=2, candidateId=leader708, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:18,775 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710: Set currentTerm=2, votedFor=leader708 12:30:18,775 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): Cluster state: [] 12:30:18,775 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader708, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:18,775 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader708, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:18,775 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: Set currentTerm=2, votedFor=null 12:30:18,775 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:18,775 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): In requestVote: RequestVote [term=2, candidateId=leader708, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:18,775 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: Set currentTerm=2, votedFor=leader708 12:30:18,775 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:18,776 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:30:18,776 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:30:18,776 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Election: Leader has following peers: [follower711, follower710, follower709] 12:30:18,776 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower711: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:18,777 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower710: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:18,777 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): Term 2 in "AppendEntries [leaderId=leader708, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:30:18,777 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower709: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:18,777 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Set currentTerm=2, votedFor=null 12:30:18,777 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:30:18,777 PM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower711: Last sync leader does not match current leader leader708, need to catch up to -1 12:30:18,777 PM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower710: Last sync leader does not match current leader leader708, need to catch up to -1 12:30:18,777 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower711 updated: matchIndex: -1, nextIndex: 0 12:30:18,777 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower711: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:18,777 PM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower709: Last sync leader does not match current leader leader708, need to catch up to -1 [WARN] [03/15/2024 12:30:18.777] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower710] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@13cfd723 12:30:18,777 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower710 updated: matchIndex: -1, nextIndex: 0 [WARN] [03/15/2024 12:30:18.777] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower709] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@56c76268 12:30:18,778 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower710: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:18,778 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower709 updated: matchIndex: -1, nextIndex: 0 12:30:18,778 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower709: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/15/2024 12:30:18.778] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower711] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@77333226 12:30:18,819 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:30:18,819 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 12:30:18,820 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 12:30:18,820 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 12:30:18,820 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:18,820 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:18,820 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 12:30:18,820 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower711: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:18,820 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): sendAppendEntries: 0 is present for follower follower711 12:30:18,821 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Sending AppendEntries to follower follower711: AppendEntries [leaderId=leader708, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:18,821 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower710: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:18,821 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): sendAppendEntries: 0 is present for follower follower710 12:30:18,821 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Sending AppendEntries to follower follower710: AppendEntries [leaderId=leader708, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:18,821 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower709: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:18,821 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): sendAppendEntries: 0 is present for follower follower709 12:30:18,821 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Sending AppendEntries to follower follower709: AppendEntries [leaderId=leader708, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:18,821 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): handleAppendEntries: AppendEntries [leaderId=leader708, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:18,821 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): Number of entries to be appended = 1 12:30:18,821 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 12:30:18,821 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): Append entry to log zero 12:30:18,821 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:18,821 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): Log size is now 1 12:30:18,821 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): handleAppendEntries: AppendEntries [leaderId=leader708, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:18,821 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): Number of entries to be appended = 1 12:30:18,821 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower711, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:30:18,821 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 12:30:18,821 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): Append entry to log zero 12:30:18,821 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower711: Lagging 0 entries behind leader leader708 and reached -1 (of expected -1) 12:30:18,821 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:18,821 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower711 updated: matchIndex: 0, nextIndex: 1 12:30:18,821 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower711: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:18,821 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): Log size is now 1 12:30:18,822 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower709, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:30:18,822 PM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower709: Lagging 0 entries behind leader leader708 and reached -1 (of expected -1) 12:30:18,822 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower709 updated: matchIndex: 0, nextIndex: 1 12:30:18,822 PM [test-akka.actor.default-dispatcher-10] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower709: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:18,823 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:18,823 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:18,823 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 12:30:18,823 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Setting last applied to 0 12:30:18,823 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Moving last applied index from -1 to 0 12:30:18,823 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Applying state for log index 0 data zero 12:30:18,823 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:30:18,823 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: applyState called: zero [WARN] [03/15/2024 12:30:18.823] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower711] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5ceb8ec0 [WARN] [03/15/2024 12:30:18.823] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower709] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3e93767b 12:30:18,823 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Persisting ApplyJournalEntries with index=0 12:30:18,893 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): sendAppendEntries: 0 is present for follower follower710 12:30:18,893 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Sending AppendEntries to follower follower710: AppendEntries [leaderId=leader708, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:19,14 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): sendAppendEntries: 0 is present for follower follower710 12:30:19,14 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): Commit index set to 0 12:30:19,14 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Sending AppendEntries to follower follower710: AppendEntries [leaderId=leader708, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:30:19,14 PM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower711: Lagging 0 entries behind leader leader708 and reached 0 (of expected -1) 12:30:19,14 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 12:30:19,14 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): Setting last applied to 0 12:30:19,14 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Moving last applied index from -1 to 0 12:30:19,14 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Applying state for log index 0 data zero 12:30:19,14 PM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: applyState called: zero 12:30:19,14 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): Commit index set to 0 12:30:19,15 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower709: Lagging 0 entries behind leader leader708 and reached 0 (of expected -1) 12:30:19,15 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 12:30:19,15 PM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Persisting ApplyJournalEntries with index=0 12:30:19,15 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709 (Follower): Setting last applied to 0 12:30:19,15 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: Moving last applied index from -1 to 0 12:30:19,15 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: Applying state for log index 0 data zero 12:30:19,15 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: applyState called: zero 12:30:19,15 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: Persisting ApplyJournalEntries with index=0 12:30:19,23 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 12:30:19,23 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 12:30:19,23 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 12:30:19,23 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: onRequestLeadership RequestLeadership [requestedFollowerId=follower710, replyTo=Actor[akka://test/user/$c#1158294826]] 12:30:19,23 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: Initiating leader transfer 12:30:19,23 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower710: onLeaderTransitioning: LeaderTransitioning [leaderId=leader708] 12:30:19,23 PM [test-akka.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader708: pauseLeader successfully completed - doing transfer 12:30:19,23 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower709: onLeaderTransitioning: LeaderTransitioning [leaderId=leader708] 12:30:19,23 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Attempting to transfer leadership 12:30:19,23 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower711: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:19,23 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: onLeaderTransitioning: LeaderTransitioning [leaderId=leader708] 12:30:19,23 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower710: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:19,24 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): sendAppendEntries: 0 is present for follower follower710 12:30:19,24 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Checking sendAppendEntries for follower follower709: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:30:19,24 PM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower709: Lagging 0 entries behind leader leader708 and reached 0 (of expected -1) 12:30:19,24 PM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower711: Lagging 0 entries behind leader leader708 and reached 0 (of expected -1) 12:30:19,134 PM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower709: Lagging 0 entries behind leader leader708 and reached 0 (of expected -1) 12:30:19,134 PM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower711: Lagging 0 entries behind leader leader708 and reached 0 (of expected -1) 12:30:19,254 PM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower709: Lagging 0 entries behind leader leader708 and reached 0 (of expected -1) 12:30:19,254 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower711: Lagging 0 entries behind leader leader708 and reached 0 (of expected -1) 12:30:19,373 PM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader708: leader transfer aborted 12:30:19,373 PM [test-akka.actor.default-dispatcher-5] [WARN] RaftActorLeadershipTransferCohort - leader708: Failed to transfer leadership in 349.5 ms 12:30:19,373 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708: LeadershipTransfer request from follower710 failed 12:30:19,373 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader708 (Leader): Leadership transfer expired 12:30:19,374 PM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower709: Lagging 0 entries behind leader leader708 and reached 0 (of expected -1) 12:30:19,374 PM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower711: Lagging 0 entries behind leader leader708 and reached 0 (of expected -1) 12:30:19,377 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 12:30:19,377 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower709-notifier712#-1883270203] 12:30:19,388 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower709-collector713#370974975] 12:30:19,388 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower709] 12:30:19,389 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower710-notifier714#-705872011] 12:30:19,389 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower710-collector715#-1181071812] 12:30:19,389 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower710] 12:30:19,389 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower711-notifier716#140630960] 12:30:19,389 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower711-collector717#1664395333] 12:30:19,390 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower711] 12:30:19,390 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader708-notifier718#387986132] 12:30:19,390 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader708-collector719#1588569338] 12:30:19,390 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader708] 12:30:19,390 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$c#1158294826] 12:30:19,391 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 12:30:19,391 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/15/2024 12:30:19.395] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower721]: org.opendaylight.controller.cluster.common.actor.Monitor@6994f823 12:30:19,395 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower721 with journal batch size 1000 12:30:19,396 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721: SnapshotOffer called. 12:30:19,396 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721: Moving last applied index from -1 to -1 12:30:19,396 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721: Set currentTerm=1, votedFor=null 12:30:19,396 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721: Updated server config: isVoting: true, peers: [PeerInfo [id=follower722, address=akka://test/user/follower722, votingState=VOTING], PeerInfo [id=leader720, address=akka://test/user/leader720, votingState=VOTING], PeerInfo [id=follower723, address=akka://test/user/follower723, votingState=NON_VOTING]] 12:30:19,396 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower721 in 87.42 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:19,397 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower721: Recovery completed in 1.012 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/15/2024 12:30:19.398] [test-akka.actor.default-dispatcher-16] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower722]: org.opendaylight.controller.cluster.common.actor.Monitor@612d6cc0 12:30:19,398 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower722 with journal batch size 1000 12:30:19,399 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722: SnapshotOffer called. 12:30:19,399 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722: Moving last applied index from -1 to -1 12:30:19,399 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722: Set currentTerm=1, votedFor=null 12:30:19,399 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722: Updated server config: isVoting: true, peers: [PeerInfo [id=follower721, address=akka://test/user/follower721, votingState=VOTING], PeerInfo [id=leader720, address=akka://test/user/leader720, votingState=VOTING], PeerInfo [id=follower723, address=akka://test/user/follower723, votingState=NON_VOTING]] 12:30:19,399 PM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower722 in 81.86 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:19,400 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower722: Recovery completed in 953.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/15/2024 12:30:19.501] [test-akka.actor.default-dispatcher-15] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower723]: org.opendaylight.controller.cluster.common.actor.Monitor@5ef28aaf 12:30:19,501 PM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower723 with journal batch size 1000 12:30:19,502 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower723: SnapshotOffer called. 12:30:19,502 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower723: Moving last applied index from -1 to -1 12:30:19,502 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower723: Set currentTerm=1, votedFor=null 12:30:19,502 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower723: Updated server config: isVoting: false, peers: [PeerInfo [id=follower722, address=akka://test/user/follower722, votingState=VOTING], PeerInfo [id=follower721, address=akka://test/user/follower721, votingState=VOTING], PeerInfo [id=leader720, address=akka://test/user/leader720, votingState=VOTING]] 12:30:19,502 PM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower723 in 87.05 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:19,503 PM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower723: Recovery completed in 1.243 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/15/2024 12:30:19.504] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader720]: org.opendaylight.controller.cluster.common.actor.Monitor@4cec0276 12:30:19,505 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader720 with journal batch size 1000 12:30:19,506 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: SnapshotOffer called. 12:30:19,506 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: Moving last applied index from -1 to -1 12:30:19,506 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: Set currentTerm=1, votedFor=null 12:30:19,506 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: Updated server config: isVoting: true, peers: [PeerInfo [id=follower722, address=akka://test/user/follower722, votingState=VOTING], PeerInfo [id=follower721, address=akka://test/user/follower721, votingState=VOTING], PeerInfo [id=follower723, address=akka://test/user/follower723, votingState=NON_VOTING]] 12:30:19,507 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader720 in 363.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:30:19,508 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: Recovery completed in 1.867 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:30:19,873 PM [test-akka.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:30:19,874 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.170.99:25520, Removed) 12:30:19,874 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Follower): Received ElectionTimeout - switching to Candidate 12:30:19,874 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Candidate): Election: Candidate has following voting peers: [follower722, follower721] 12:30:19,874 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: Set currentTerm=2, votedFor=leader720 12:30:19,874 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Candidate): Starting new election term 2 12:30:19,874 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Candidate): Sending RequestVote [term=2, candidateId=leader720, lastLogIndex=-1, lastLogTerm=-1] to peer follower722 12:30:19,874 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Candidate): Sending RequestVote [term=2, candidateId=leader720, lastLogIndex=-1, lastLogTerm=-1] to peer follower721 12:30:19,874 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:30:19,874 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:19,874 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:19,874 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721 (Follower): Cluster state: [] 12:30:19,874 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader720, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:19,874 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower721 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader720, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:19,874 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721: Set currentTerm=2, votedFor=null 12:30:19,874 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Follower): Cluster state: [] 12:30:19,874 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader720, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:19,874 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721 (Follower): In requestVote: RequestVote [term=2, candidateId=leader720, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:19,874 PM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader720, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:30:19,874 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721: Set currentTerm=2, votedFor=leader720 12:30:19,874 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722: Set currentTerm=2, votedFor=null 12:30:19,875 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower721 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:19,875 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Follower): In requestVote: RequestVote [term=2, candidateId=leader720, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:19,875 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722: Set currentTerm=2, votedFor=leader720 12:30:19,875 PM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:30:19,876 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:30:19,876 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:30:19,876 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Election: Leader has following peers: [follower722, follower721, follower723] 12:30:19,877 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower722: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,877 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower721: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,877 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower723: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,877 PM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower722: Last sync leader does not match current leader leader720, need to catch up to -1 12:30:19,877 PM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:30:19,877 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower721: Last sync leader does not match current leader leader720, need to catch up to -1 12:30:19,877 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower723 (Follower): Term 2 in "AppendEntries [leaderId=leader720, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:30:19,877 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower723: Set currentTerm=2, votedFor=null [WARN] [03/15/2024 12:30:19.877] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower722] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3fff9082 12:30:19,877 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower723: Last sync leader does not match current leader leader720, need to catch up to -1 [WARN] [03/15/2024 12:30:19.877] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower721] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7c774043 12:30:19,877 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower722 updated: matchIndex: -1, nextIndex: 0 12:30:19,877 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower722: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,877 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower721 updated: matchIndex: -1, nextIndex: 0 12:30:19,878 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower721: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,878 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower723 updated: matchIndex: -1, nextIndex: 0 12:30:19,878 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower723: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/15/2024 12:30:19.878] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower723] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7d33380e 12:30:19,912 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:30:19,912 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 12:30:19,913 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 12:30:19,913 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower721] 12:30:19,924 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower723] 12:30:19,935 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 12:30:19,935 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 12:30:19,935 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: onRequestLeadership RequestLeadership [requestedFollowerId=follower722, replyTo=Actor[akka://test/user/$d#139639478]] 12:30:19,935 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: Initiating leader transfer 12:30:19,936 PM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722: onLeaderTransitioning: LeaderTransitioning [leaderId=leader720] [WARN] [03/15/2024 12:30:19.936] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower721] received dead letter from TestActor[akka://test/user/leader720]: LeaderTransitioning [leaderId=leader720] 12:30:19,936 PM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader720: pauseLeader successfully completed - doing transfer [WARN] [03/15/2024 12:30:19.936] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower723] received dead letter from TestActor[akka://test/user/leader720]: LeaderTransitioning [leaderId=leader720] 12:30:19,936 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Attempting to transfer leadership 12:30:19,936 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower722: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,936 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower721: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,936 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower723: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,936 PM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower722: Lagging 0 entries behind leader leader720 and reached -1 (of expected -1) [WARN] [03/15/2024 12:30:19.936] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower721] received dead letter from TestActor[akka://test/user/leader720]: AppendEntries [leaderId=leader720, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [03/15/2024 12:30:19.936] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower723] received dead letter from TestActor[akka://test/user/leader720]: AppendEntries [leaderId=leader720, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [03/15/2024 12:30:19.936] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower722] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3c392960 12:30:19,937 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): tryToCompleteLeadershipTransfer: followerId: follower722, matchIndex: -1, lastIndex: -1, isVoting: true 12:30:19,937 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Follower's log matches - sending ElectionTimeout 12:30:19,937 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower722: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,937 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower721: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,937 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower722: Lagging 0 entries behind leader leader720 and reached -1 (of expected -1) 12:30:19,937 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Checking sendAppendEntries for follower follower723: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/15/2024 12:30:19.937] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower721] received dead letter from TestActor[akka://test/user/leader720]: AppendEntries [leaderId=leader720, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [03/15/2024 12:30:19.937] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower723] received dead letter from TestActor[akka://test/user/leader720]: AppendEntries [leaderId=leader720, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:30:19,937 PM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Leader transfer complete 12:30:19,937 PM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader720: leader transfer complete - waiting for new leader 12:30:19,937 PM [test-akka.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:30:19,937 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Follower): Received TimeoutNow - switching to Candidate 12:30:19,937 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Candidate): Election: Candidate has following voting peers: [follower721, leader720] 12:30:19,937 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722: Set currentTerm=3, votedFor=follower722 12:30:19,937 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Candidate): Starting new election term 3 12:30:19,937 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Candidate): Sending RequestVote [term=3, candidateId=follower722, lastLogIndex=-1, lastLogTerm=-1] to peer follower721 12:30:19,938 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Candidate): Sending RequestVote [term=3, candidateId=follower722, lastLogIndex=-1, lastLogTerm=-1] to peer leader720 12:30:19,938 PM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 [WARN] [03/15/2024 12:30:19.938] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower721] received dead letter from TestActor[akka://test/user/follower722]: RequestVote [term=3, candidateId=follower722, lastLogIndex=-1, lastLogTerm=-1] 12:30:19,938 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:30:19,938 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Cluster state: [] 12:30:19,938 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower722, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:30:19,938 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower722, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 12:30:19,938 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: Set currentTerm=3, votedFor=null 12:30:19,938 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): Leadership transfer in progress - processing RequestVote 12:30:19,938 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): In requestVote: RequestVote [term=3, candidateId=follower722, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 12:30:19,938 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: Set currentTerm=3, votedFor=follower722 12:30:19,938 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 12:30:19,938 PM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader720 (Leader) :- Switching from behavior Leader to Follower, election term: 3 12:30:19,939 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 12:30:19,939 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 12:30:19,939 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Leader): Election: Leader has following peers: [follower721, leader720, follower723] 12:30:19,939 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Leader): Checking sendAppendEntries for follower follower721: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,939 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Leader): Checking sendAppendEntries for follower leader720: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,939 PM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Leader): Checking sendAppendEntries for follower follower723: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,939 PM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [03/15/2024 12:30:19.939] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower721] received dead letter from TestActor[akka://test/user/follower722]: AppendEntries [leaderId=follower722, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [03/15/2024 12:30:19.939] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower723] received dead letter from TestActor[akka://test/user/follower722]: AppendEntries [leaderId=follower722, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:30:19,940 PM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - leader720: Last sync leader does not match current leader follower722, need to catch up to -1 12:30:19,940 PM [test-akka.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader720: leader changed to follower722 12:30:19,940 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader720 updated: matchIndex: -1, nextIndex: 0 12:30:19,940 PM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower722 (Leader): Checking sendAppendEntries for follower leader720: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:30:19,940 PM [test-akka.actor.default-dispatcher-15] [INFO] RaftActorLeadershipTransferCohort - leader720: Successfully transferred leadership to follower722 in 4.341 ms 12:30:19,940 PM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader720: Leadership transferred successfully to follower722 [WARN] [03/15/2024 12:30:19.940] [test-akka.actor.default-dispatcher-6] [akka://test/user/leader720] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@20f1632c 12:30:19,987 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 12:30:19,987 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower721-notifier724#413191672] 12:30:19,998 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower721-collector725#-231929807] 12:30:19,999 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower722-notifier726#-636200554] 12:30:19,999 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower722-collector727#1801900454] 12:30:19,999 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower722] 12:30:19,999 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower723-notifier728#-1613433797] 12:30:19,999 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower723-collector729#-766508085] 12:30:19,999 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader720-notifier730#142994084] 12:30:20,0 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader720-collector731#-221900231] 12:30:20,0 PM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader720] 12:30:20,0 PM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$d#139639478] 12:30:20,1 PM [test-akka.actor.default-dispatcher-17] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [INFO] [03/15/2024 12:30:20.001] [main] [CoordinatedShutdown(akka://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [03/15/2024 12:30:20.003] [test-akka.actor.default-dispatcher-10] [akka://test/system/cluster/core/daemon] unhandled message from Actor[akka://test/deadLetters]: Leave(akka://test@10.30.170.99:25520) [INFO] [03/15/2024 12:30:20.003] [test-akka.actor.internal-dispatcher-11] [Cluster(akka://test)] Cluster Node [akka://test@10.30.170.99:25520] - Exiting completed 12:30:20,4 PM [test-akka.actor.default-dispatcher-17] [INFO] Cluster - Cluster Node [akka://test@10.30.170.99:25520] - Exiting completed [INFO] [03/15/2024 12:30:20.004] [test-akka.actor.internal-dispatcher-11] [Cluster(akka://test)] Cluster Node [akka://test@10.30.170.99:25520] - Shutting down... 12:30:20,4 PM [test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://test@10.30.170.99:25520] - Shutting down... [INFO] [03/15/2024 12:30:20.004] [test-akka.actor.internal-dispatcher-11] [Cluster(akka://test)] Cluster Node [akka://test@10.30.170.99:25520] - Successfully shut down 12:30:20,4 PM [test-akka.actor.default-dispatcher-10] [INFO] Cluster - Cluster Node [akka://test@10.30.170.99:25520] - Successfully shut down [INFO] [03/15/2024 12:30:20.009] [test-akka.remote.default-remote-dispatcher-8] [akka://test@10.30.170.99:25520/system/remoting-terminator] Shutting down remote daemon. 12:30:20,9 PM [test-akka.actor.default-dispatcher-17] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. [INFO] [03/15/2024 12:30:20.009] [test-akka.remote.default-remote-dispatcher-8] [akka://test@10.30.170.99:25520/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 12:30:20,9 PM [test-akka.actor.default-dispatcher-10] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [03/15/2024 12:30:20.012] [test-akka.remote.default-remote-dispatcher-7] [akka://test@10.30.170.99:25520/system/remoting-terminator] Remoting shut down. 12:30:20,13 PM [test-akka.actor.default-dispatcher-17] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [DEBUG] [03/15/2024 12:30:20.013] [test-akka.actor.internal-dispatcher-2] [EventStream] shutting down: StandardOutLogger