[DEBUG] [01/22/2024 06:56:28.213] [main] [EventStream(akka://test)] logger log1-TestEventListener started 06:56:28,216 AM [test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [01/22/2024 06:56:28.216] [main] [EventStream(akka://test)] logger log2-Slf4jLogger started [DEBUG] [01/22/2024 06:56:28.216] [main] [EventStream(akka://test)] Default Loggers started [INFO] [01/22/2024 06:56:28.234] [main] [ArteryTransport(akka://test)] Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.171.69:25520] with UID [-527390118260691748] [INFO] [01/22/2024 06:56:28.235] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.69:25520] - Starting up, Akka version [2.6.21] ... 06:56:28,236 AM [test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.171.69:25520] with UID [-527390118260691748] 06:56:28,236 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.69:25520] - Starting up, Akka version [2.6.21] ... [INFO] [01/22/2024 06:56:28.245] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.69:25520] - Registered cluster JMX MBean [akka:type=Cluster] [INFO] [01/22/2024 06:56:28.245] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.69:25520] - Started up successfully 06:56:28,245 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@10.30.171.69:25520] - Registered cluster JMX MBean [akka:type=Cluster] 06:56:28,245 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@10.30.171.69:25520] - Started up successfully [INFO] [01/22/2024 06:56:28.247] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.69:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing [INFO] [01/22/2024 06:56:28.247] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.69:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 06:56:28,248 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@10.30.171.69:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 06:56:28,248 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@10.30.171.69:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 06:56:28,260 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 06:56:28,260 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 06:56:28,278 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower442 with journal batch size 1000 06:56:28,278 AM [test-akka.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower442] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [01/22/2024 06:56:28.284] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower442]: org.opendaylight.controller.cluster.common.actor.Monitor@40e5b21c [INFO] [akkaDeadLetter][01/22/2024 06:56:28.278] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower442] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,292 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442: SnapshotOffer called. 06:56:28,293 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442: Moving last applied index from -1 to -1 06:56:28,293 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442: Set currentTerm=1, votedFor=null 06:56:28,293 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442: Updated server config: isVoting: true, peers: [PeerInfo [id=leader441, address=akka://test/user/leader441, votingState=VOTING], PeerInfo [id=follower443, address=akka://test/user/follower443, votingState=VOTING], PeerInfo [id=follower444, address=akka://test/user/follower444, votingState=NON_VOTING]] 06:56:28,293 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower442 in 157.6 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:28,294 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower442: Recovery completed in 1.393 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:28.377] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower443]: org.opendaylight.controller.cluster.common.actor.Monitor@34a8b1b6 [INFO] [akkaDeadLetter][01/22/2024 06:56:28.377] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower443] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,377 AM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower443] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,378 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower443 with journal batch size 1000 06:56:28,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443: SnapshotOffer called. 06:56:28,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443: Moving last applied index from -1 to -1 06:56:28,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443: Set currentTerm=1, votedFor=null 06:56:28,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443: Updated server config: isVoting: true, peers: [PeerInfo [id=leader441, address=akka://test/user/leader441, votingState=VOTING], PeerInfo [id=follower442, address=akka://test/user/follower442, votingState=VOTING], PeerInfo [id=follower444, address=akka://test/user/follower444, votingState=NON_VOTING]] 06:56:28,379 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower443 in 91.06 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:28,380 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower443: Recovery completed in 1.421 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:28.479] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower444]: org.opendaylight.controller.cluster.common.actor.Monitor@d1f6119 [INFO] [akkaDeadLetter][01/22/2024 06:56:28.480] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower444] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,480 AM [test-akka.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower444] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,480 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower444 with journal batch size 1000 06:56:28,481 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower444: SnapshotOffer called. 06:56:28,481 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower444: Moving last applied index from -1 to -1 06:56:28,481 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower444: Set currentTerm=1, votedFor=null 06:56:28,481 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower444: Updated server config: isVoting: false, peers: [PeerInfo [id=leader441, address=akka://test/user/leader441, votingState=VOTING], PeerInfo [id=follower443, address=akka://test/user/follower443, votingState=VOTING], PeerInfo [id=follower442, address=akka://test/user/follower442, votingState=VOTING]] 06:56:28,482 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower444 in 107.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:28,483 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower444: Recovery completed in 1.143 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:28.581] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader441]: org.opendaylight.controller.cluster.common.actor.Monitor@6f883bf1 [INFO] [akkaDeadLetter][01/22/2024 06:56:28.581] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader441] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,581 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader441] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,582 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader441 with journal batch size 1000 06:56:28,583 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441: SnapshotOffer called. 06:56:28,583 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441: Moving last applied index from -1 to -1 06:56:28,583 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441: Set currentTerm=1, votedFor=null 06:56:28,583 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441: Updated server config: isVoting: true, peers: [PeerInfo [id=follower443, address=akka://test/user/follower443, votingState=VOTING], PeerInfo [id=follower442, address=akka://test/user/follower442, votingState=VOTING], PeerInfo [id=follower444, address=akka://test/user/follower444, votingState=NON_VOTING]] 06:56:28,583 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader441 in 112.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:28,584 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader441: Recovery completed in 1.216 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 06:56:28,914 AM [test-akka.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 06:56:28,914 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.69:25520, Removed) 06:56:28,914 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Follower): Received ElectionTimeout - switching to Candidate 06:56:28,915 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Candidate): Election: Candidate has following voting peers: [follower443, follower442] 06:56:28,915 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441: Set currentTerm=2, votedFor=leader441 06:56:28,915 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Candidate): Starting new election term 2 06:56:28,915 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Candidate): Sending RequestVote [term=2, candidateId=leader441, lastLogIndex=-1, lastLogTerm=-1] to peer follower443 06:56:28,915 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Candidate): Sending RequestVote [term=2, candidateId=leader441, lastLogIndex=-1, lastLogTerm=-1] to peer follower442 06:56:28,915 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:28,915 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 06:56:28,915 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443 (Follower): Cluster state: [] 06:56:28,915 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader441, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:28,915 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower443 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader441, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:28,915 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443: Set currentTerm=2, votedFor=null 06:56:28,915 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443 (Follower): In requestVote: RequestVote [term=2, candidateId=leader441, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:28,916 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443: Set currentTerm=2, votedFor=leader441 06:56:28,916 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:28,915 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:28,916 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442 (Follower): Cluster state: [] 06:56:28,916 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader441, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:28,916 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower442 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader441, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:28,916 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442: Set currentTerm=2, votedFor=null 06:56:28,916 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442 (Follower): In requestVote: RequestVote [term=2, candidateId=leader441, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:28,916 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442: Set currentTerm=2, votedFor=leader441 06:56:28,916 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:28,916 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 06:56:28,916 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 06:56:28,916 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Election: Leader has following peers: [follower443, follower442, follower444] 06:56:28,916 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Checking sendAppendEntries for follower follower443: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:28,917 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Checking sendAppendEntries for follower follower442: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:28,917 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Checking sendAppendEntries for follower follower444: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:28,917 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower443: Last sync leader does not match current leader leader441, need to catch up to -1 06:56:28,917 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 06:56:28,917 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower443 updated: matchIndex: -1, nextIndex: 0 06:56:28,917 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Checking sendAppendEntries for follower follower443: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:28.917] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower443] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@55b2d124 06:56:28,917 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower444 (Follower): Term 2 in "AppendEntries [leaderId=leader441, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term [INFO] [akkaDeadLetter][01/22/2024 06:56:28.917] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower443] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower443] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,917 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower444: Set currentTerm=2, votedFor=null 06:56:28,917 AM [test-akka.actor.default-dispatcher-5] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower443] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,917 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower444: Last sync leader does not match current leader leader441, need to catch up to -1 06:56:28,918 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower444 updated: matchIndex: -1, nextIndex: 0 06:56:28,918 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Checking sendAppendEntries for follower follower444: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:28,918 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower442: Last sync leader does not match current leader leader441, need to catch up to -1 06:56:28,918 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower442 updated: matchIndex: -1, nextIndex: 0 [WARN] [01/22/2024 06:56:28.918] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower442] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6e8e29ac 06:56:28,919 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Checking sendAppendEntries for follower follower442: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [INFO] [akkaDeadLetter][01/22/2024 06:56:28.919] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower442] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower442] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [01/22/2024 06:56:28.919] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower444] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1f5a00ca 06:56:28,919 AM [test-akka.actor.default-dispatcher-15] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower442] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][01/22/2024 06:56:28.919] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower444] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower444] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,919 AM [test-akka.actor.default-dispatcher-15] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower444] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,933 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 06:56:28,933 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader441 starting 06:56:28,935 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441: onShutDown 06:56:28,935 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441: Initiating leader transfer 06:56:28,936 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower443: onLeaderTransitioning: LeaderTransitioning [leaderId=leader441] 06:56:28,936 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower444: onLeaderTransitioning: LeaderTransitioning [leaderId=leader441] 06:56:28,936 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower442: onLeaderTransitioning: LeaderTransitioning [leaderId=leader441] 06:56:28,936 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader441: pauseLeader successfully completed - doing transfer 06:56:28,936 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Attempting to transfer leadership 06:56:28,936 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Checking sendAppendEntries for follower follower443: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:28,936 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Checking sendAppendEntries for follower follower442: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:28,936 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Checking sendAppendEntries for follower follower444: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:28,936 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower442: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) 06:56:28,936 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower443: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) 06:56:28,936 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower444: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) [WARN] [01/22/2024 06:56:28.937] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower442] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4254573a 06:56:28,937 AM [test-akka.actor.default-dispatcher-16] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower442] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:28,937 AM [test-akka.actor.default-dispatcher-16] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower443] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [01/22/2024 06:56:28.937] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower443] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4c93a100 [WARN] [01/22/2024 06:56:28.937] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower444] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@48ac3c4a 06:56:28,937 AM [test-akka.actor.default-dispatcher-17] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower444] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][01/22/2024 06:56:28.937] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower442] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower442] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][01/22/2024 06:56:28.937] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower443] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower443] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][01/22/2024 06:56:28.937] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower444] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower444] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 06:56:29,35 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower444: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) 06:56:29,35 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower442: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) 06:56:29,35 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower443: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) 06:56:29,155 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower443: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) 06:56:29,155 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower444: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) 06:56:29,155 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower442: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) 06:56:29,274 AM [test-akka.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader441: leader transfer aborted 06:56:29,275 AM [test-akka.actor.default-dispatcher-11] [WARN] RaftActorLeadershipTransferCohort - leader441: Failed to transfer leadership in 339.0 ms 06:56:29,275 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441: leader transfer failed - sending PoisonPill 06:56:29,275 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader441 (Leader): Leadership transfer expired 06:56:29,275 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower442: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) 06:56:29,275 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower443: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) [WARN] [01/22/2024 06:56:29.275] [test-akka.actor.default-dispatcher-5] [akka://test/user/leader441] received dead letter from TestActor[akka://test/user/follower442]: AppendEntriesReply [term=2, success=true, followerId=follower442, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 06:56:29,276 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower444: Lagging 0 entries behind leader leader441 and reached -1 (of expected -1) [WARN] [01/22/2024 06:56:29.276] [test-akka.actor.default-dispatcher-5] [akka://test/user/leader441] received dead letter from TestActor[akka://test/user/follower443]: AppendEntriesReply [term=2, success=true, followerId=follower443, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [01/22/2024 06:56:29.276] [test-akka.actor.default-dispatcher-5] [akka://test/user/leader441] received dead letter from TestActor[akka://test/user/follower444]: AppendEntriesReply [term=2, success=true, followerId=follower444, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 06:56:29,277 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader441 ending 06:56:29,279 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 06:56:29,279 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower442-notifier445#1153325333] 06:56:29,291 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower442-collector446#134263202] 06:56:29,292 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower442] 06:56:29,292 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower443-notifier447#-693011099] 06:56:29,293 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower443-collector448#662353047] 06:56:29,293 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower443] 06:56:29,293 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower444-notifier449#733342241] 06:56:29,293 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower444-collector450#1988943544] 06:56:29,293 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower444] 06:56:29,294 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader441-notifier451#-494125949] 06:56:29,294 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader441-collector452#1008293200] 06:56:29,294 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader441] 06:56:29,295 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 06:56:29,295 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/22/2024 06:56:29.298] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower454]: org.opendaylight.controller.cluster.common.actor.Monitor@4c87e3a6 06:56:29,299 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower454 with journal batch size 1000 06:56:29,300 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454: SnapshotOffer called. 06:56:29,300 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454: Moving last applied index from -1 to -1 06:56:29,300 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454: Set currentTerm=1, votedFor=null 06:56:29,300 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454: Updated server config: isVoting: true, peers: [PeerInfo [id=leader453, address=akka://test/user/leader453, votingState=VOTING], PeerInfo [id=follower456, address=akka://test/user/follower456, votingState=NON_VOTING], PeerInfo [id=follower455, address=akka://test/user/follower455, votingState=VOTING]] 06:56:29,300 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower454 in 110.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:29,303 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower454: Recovery completed in 3.109 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:29.400] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower455]: org.opendaylight.controller.cluster.common.actor.Monitor@49e50048 06:56:29,400 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower455 with journal batch size 1000 06:56:29,401 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455: SnapshotOffer called. 06:56:29,401 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455: Moving last applied index from -1 to -1 06:56:29,401 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455: Set currentTerm=1, votedFor=null 06:56:29,401 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455: Updated server config: isVoting: true, peers: [PeerInfo [id=leader453, address=akka://test/user/leader453, votingState=VOTING], PeerInfo [id=follower454, address=akka://test/user/follower454, votingState=VOTING], PeerInfo [id=follower456, address=akka://test/user/follower456, votingState=NON_VOTING]] 06:56:29,401 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower455 in 90.64 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:29,403 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower455: Recovery completed in 1.405 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:29.502] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower456]: org.opendaylight.controller.cluster.common.actor.Monitor@1b35c643 06:56:29,503 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower456 with journal batch size 1000 06:56:29,504 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower456: SnapshotOffer called. 06:56:29,504 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower456: Moving last applied index from -1 to -1 06:56:29,504 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower456: Set currentTerm=1, votedFor=null 06:56:29,504 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower456: Updated server config: isVoting: false, peers: [PeerInfo [id=leader453, address=akka://test/user/leader453, votingState=VOTING], PeerInfo [id=follower454, address=akka://test/user/follower454, votingState=VOTING], PeerInfo [id=follower455, address=akka://test/user/follower455, votingState=VOTING]] 06:56:29,504 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower456 in 127.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:29,505 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower456: Recovery completed in 1.606 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:29.605] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader453]: org.opendaylight.controller.cluster.common.actor.Monitor@61fdf26a 06:56:29,605 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader453 with journal batch size 1000 06:56:29,608 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: SnapshotOffer called. 06:56:29,608 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: Moving last applied index from -1 to -1 06:56:29,608 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: Set currentTerm=1, votedFor=null 06:56:29,608 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: Updated server config: isVoting: true, peers: [PeerInfo [id=follower454, address=akka://test/user/follower454, votingState=VOTING], PeerInfo [id=follower456, address=akka://test/user/follower456, votingState=NON_VOTING], PeerInfo [id=follower455, address=akka://test/user/follower455, votingState=VOTING]] 06:56:29,608 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader453 in 64.54 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:29,611 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: Recovery completed in 2.373 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 06:56:29,934 AM [test-akka.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 06:56:29,934 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.69:25520, Removed) 06:56:29,934 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Follower): Received ElectionTimeout - switching to Candidate 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Candidate): Election: Candidate has following voting peers: [follower454, follower455] 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: Set currentTerm=2, votedFor=leader453 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Candidate): Starting new election term 2 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Candidate): Sending RequestVote [term=2, candidateId=leader453, lastLogIndex=-1, lastLogTerm=-1] to peer follower454 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Candidate): Sending RequestVote [term=2, candidateId=leader453, lastLogIndex=-1, lastLogTerm=-1] to peer follower455 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 06:56:29,935 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:29,935 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): Cluster state: [] 06:56:29,935 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader453, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:29,935 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader453, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:29,935 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454: Set currentTerm=2, votedFor=null 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Follower): Cluster state: [] 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader453, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:29,935 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): In requestVote: RequestVote [term=2, candidateId=leader453, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader453, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:29,935 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454: Set currentTerm=2, votedFor=leader453 06:56:29,935 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455: Set currentTerm=2, votedFor=null 06:56:29,935 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:29,936 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Follower): In requestVote: RequestVote [term=2, candidateId=leader453, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:29,936 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455: Set currentTerm=2, votedFor=leader453 06:56:29,936 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:29,937 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 06:56:29,937 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 06:56:29,937 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Election: Leader has following peers: [follower454, follower456, follower455] 06:56:29,937 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower454: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,937 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower456: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,937 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower455: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,938 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 06:56:29,938 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower454: Last sync leader does not match current leader leader453, need to catch up to -1 06:56:29,938 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower456 (Follower): Term 2 in "AppendEntries [leaderId=leader453, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 06:56:29,938 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower456: Set currentTerm=2, votedFor=null 06:56:29,938 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower456: Last sync leader does not match current leader leader453, need to catch up to -1 06:56:29,938 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower454 updated: matchIndex: -1, nextIndex: 0 06:56:29,938 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower454: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,938 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower456 updated: matchIndex: -1, nextIndex: 0 06:56:29,938 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower456: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:29.938] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower454] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3a000475 06:56:29,939 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower455: Last sync leader does not match current leader leader453, need to catch up to -1 06:56:29,939 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower455 updated: matchIndex: -1, nextIndex: 0 06:56:29,939 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower455: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:29.939] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower455] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6cb5c4c9 [WARN] [01/22/2024 06:56:29.939] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower456] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1f32e48b 06:56:29,955 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 06:56:29,955 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 06:56:29,956 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 06:56:29,956 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 06:56:29,956 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 06:56:29,957 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: onRequestLeadership RequestLeadership [requestedFollowerId=follower455, replyTo=Actor[akka://test/user/$a#524288170]] 06:56:29,957 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: Initiating leader transfer 06:56:29,958 AM [test-akka.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader453: pauseLeader successfully completed - doing transfer 06:56:29,958 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Attempting to transfer leadership 06:56:29,958 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower454: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,958 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454: onLeaderTransitioning: LeaderTransitioning [leaderId=leader453] 06:56:29,958 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower456: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,958 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower455: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,958 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower456: onLeaderTransitioning: LeaderTransitioning [leaderId=leader453] 06:56:29,958 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower454: Lagging 0 entries behind leader leader453 and reached -1 (of expected -1) 06:56:29,958 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower456: Lagging 0 entries behind leader leader453 and reached -1 (of expected -1) 06:56:29,958 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower455: Lagging 0 entries behind leader leader453 and reached -1 (of expected -1) 06:56:29,958 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455: onLeaderTransitioning: LeaderTransitioning [leaderId=leader453] [WARN] [01/22/2024 06:56:29.958] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower454] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5116b37b [WARN] [01/22/2024 06:56:29.958] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower455] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@20dd1c6 [WARN] [01/22/2024 06:56:29.958] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower456] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@30ae0670 06:56:29,958 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): tryToCompleteLeadershipTransfer: followerId: follower455, matchIndex: -1, lastIndex: -1, isVoting: true 06:56:29,958 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Follower's log matches - sending ElectionTimeout 06:56:29,958 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower454: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,959 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower456: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,959 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Checking sendAppendEntries for follower follower455: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,959 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Leader transfer complete 06:56:29,959 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower454: Lagging 0 entries behind leader leader453 and reached -1 (of expected -1) 06:56:29,959 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader453: leader transfer complete - waiting for new leader 06:56:29,959 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower456: Lagging 0 entries behind leader leader453 and reached -1 (of expected -1) 06:56:29,959 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower455: Lagging 0 entries behind leader leader453 and reached -1 (of expected -1) 06:56:29,959 AM [test-akka.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 06:56:29,959 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Follower): Received TimeoutNow - switching to Candidate 06:56:29,959 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Candidate): Election: Candidate has following voting peers: [leader453, follower454] 06:56:29,959 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455: Set currentTerm=3, votedFor=follower455 06:56:29,959 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Candidate): Starting new election term 3 06:56:29,959 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Candidate): Sending RequestVote [term=3, candidateId=follower455, lastLogIndex=-1, lastLogTerm=-1] to peer leader453 06:56:29,960 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Candidate): Sending RequestVote [term=3, candidateId=follower455, lastLogIndex=-1, lastLogTerm=-1] to peer follower454 06:56:29,960 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 06:56:29,960 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:29,960 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Cluster state: [] 06:56:29,960 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower455, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:29,960 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower455, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 06:56:29,960 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: Set currentTerm=3, votedFor=null 06:56:29,960 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): Leadership transfer in progress - processing RequestVote 06:56:29,960 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): In requestVote: RequestVote [term=3, candidateId=follower455, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:29,960 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: Set currentTerm=3, votedFor=follower455 06:56:29,960 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 06:56:29,960 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:29,960 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): Cluster state: [] 06:56:29,960 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower455, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:29,960 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower455, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 2 - updating term 06:56:29,960 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454: Set currentTerm=3, votedFor=null 06:56:29,960 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader453 (Leader) :- Switching from behavior Leader to Follower, election term: 3 06:56:29,960 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): In requestVote: RequestVote [term=3, candidateId=follower455, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:29,960 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454: Set currentTerm=3, votedFor=follower455 06:56:29,960 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower454 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 06:56:29,963 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 06:56:29,963 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 06:56:29,963 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Leader): Election: Leader has following peers: [leader453, follower454, follower456] 06:56:29,963 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Leader): Checking sendAppendEntries for follower leader453: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,963 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Leader): Checking sendAppendEntries for follower follower454: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,963 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Leader): Checking sendAppendEntries for follower follower456: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,963 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 06:56:29,964 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower454: Last sync leader does not match current leader follower455, need to catch up to -1 06:56:29,964 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower456 (Follower): Term 3 in "AppendEntries [leaderId=follower455, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 06:56:29,964 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower456: Set currentTerm=3, votedFor=null 06:56:29,964 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - leader453: Last sync leader does not match current leader follower455, need to catch up to -1 06:56:29,964 AM [test-akka.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader453: leader changed to follower455 06:56:29,964 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower456: Last sync leader does not match current leader follower455, need to catch up to -1 06:56:29,964 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower454 updated: matchIndex: -1, nextIndex: 0 06:56:29,964 AM [test-akka.actor.default-dispatcher-12] [INFO] RaftActorLeadershipTransferCohort - leader453: Successfully transferred leadership to follower455 in 6.510 ms 06:56:29,964 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Leader): Checking sendAppendEntries for follower follower454: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,964 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader453: Leadership transferred successfully to follower455 06:56:29,964 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader453 updated: matchIndex: -1, nextIndex: 0 06:56:29,964 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Leader): Checking sendAppendEntries for follower leader453: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:29,964 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower456 updated: matchIndex: -1, nextIndex: 0 [WARN] [01/22/2024 06:56:29.964] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower454] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5541c388 06:56:29,965 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower455 (Leader): Checking sendAppendEntries for follower follower456: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:29.965] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader453] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1b385471 [WARN] [01/22/2024 06:56:29.966] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower456] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@ab78fbc 06:56:30,32 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 06:56:30,32 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower454-notifier457#1087173993] 06:56:30,43 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower454-collector458#709126010] 06:56:30,43 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower454] 06:56:30,43 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower455-notifier459#-697777676] 06:56:30,44 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower455-collector460#-647302319] 06:56:30,44 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower455] 06:56:30,44 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower456-notifier461#1848731066] 06:56:30,44 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower456-collector462#-1857266203] 06:56:30,45 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower456] 06:56:30,45 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader453-notifier463#1639068749] 06:56:30,45 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader453-collector464#-980661919] 06:56:30,45 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader453] 06:56:30,46 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$a#524288170] 06:56:30,47 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 06:56:30,47 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/22/2024 06:56:30.053] [test-akka.actor.default-dispatcher-16] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower466]: org.opendaylight.controller.cluster.common.actor.Monitor@1c89e7ae 06:56:30,55 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower466 with journal batch size 1000 06:56:30,55 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466: SnapshotOffer called. 06:56:30,55 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466: Moving last applied index from -1 to -1 06:56:30,55 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466: Set currentTerm=1, votedFor=null 06:56:30,55 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466: Updated server config: isVoting: true, peers: [PeerInfo [id=leader465, address=akka://test/user/leader465, votingState=VOTING], PeerInfo [id=follower468, address=akka://test/user/follower468, votingState=NON_VOTING], PeerInfo [id=follower467, address=akka://test/user/follower467, votingState=VOTING]] 06:56:30,56 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower466 in 71.49 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:30,56 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower466: Recovery completed in 843.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:30.156] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower467]: org.opendaylight.controller.cluster.common.actor.Monitor@d43232d 06:56:30,157 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower467 with journal batch size 1000 06:56:30,157 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467: SnapshotOffer called. 06:56:30,157 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467: Moving last applied index from -1 to -1 06:56:30,157 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467: Set currentTerm=1, votedFor=null 06:56:30,157 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467: Updated server config: isVoting: true, peers: [PeerInfo [id=leader465, address=akka://test/user/leader465, votingState=VOTING], PeerInfo [id=follower468, address=akka://test/user/follower468, votingState=NON_VOTING], PeerInfo [id=follower466, address=akka://test/user/follower466, votingState=VOTING]] 06:56:30,158 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower467 in 105.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:30,159 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower467: Recovery completed in 1.369 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:30.258] [test-akka.actor.default-dispatcher-16] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower468]: org.opendaylight.controller.cluster.common.actor.Monitor@3d399e5 06:56:30,259 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower468 with journal batch size 1000 06:56:30,260 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower468: SnapshotOffer called. 06:56:30,260 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower468: Moving last applied index from -1 to -1 06:56:30,260 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower468: Set currentTerm=1, votedFor=null 06:56:30,260 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower468: Updated server config: isVoting: false, peers: [PeerInfo [id=leader465, address=akka://test/user/leader465, votingState=VOTING], PeerInfo [id=follower467, address=akka://test/user/follower467, votingState=VOTING], PeerInfo [id=follower466, address=akka://test/user/follower466, votingState=VOTING]] 06:56:30,260 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower468 in 109.4 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:30,261 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower468: Recovery completed in 1.301 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:30.360] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader465]: org.opendaylight.controller.cluster.common.actor.Monitor@3fa9b43d 06:56:30,361 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader465 with journal batch size 1000 06:56:30,362 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465: SnapshotOffer called. 06:56:30,362 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465: Moving last applied index from -1 to -1 06:56:30,362 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465: Set currentTerm=1, votedFor=null 06:56:30,362 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465: Updated server config: isVoting: true, peers: [PeerInfo [id=follower468, address=akka://test/user/follower468, votingState=NON_VOTING], PeerInfo [id=follower467, address=akka://test/user/follower467, votingState=VOTING], PeerInfo [id=follower466, address=akka://test/user/follower466, votingState=VOTING]] 06:56:30,362 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader465 in 113.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:30,363 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader465: Recovery completed in 1.314 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 06:56:30,684 AM [test-akka.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 06:56:30,684 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.69:25520, Removed) 06:56:30,684 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Follower): Received ElectionTimeout - switching to Candidate 06:56:30,685 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Candidate): Election: Candidate has following voting peers: [follower467, follower466] 06:56:30,685 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465: Set currentTerm=2, votedFor=leader465 06:56:30,685 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Candidate): Starting new election term 2 06:56:30,685 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Candidate): Sending RequestVote [term=2, candidateId=leader465, lastLogIndex=-1, lastLogTerm=-1] to peer follower467 06:56:30,685 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Candidate): Sending RequestVote [term=2, candidateId=leader465, lastLogIndex=-1, lastLogTerm=-1] to peer follower466 06:56:30,685 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 06:56:30,685 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:30,685 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467 (Follower): Cluster state: [] 06:56:30,685 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:30,685 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader465, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:30,685 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower467 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader465, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:30,685 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466 (Follower): Cluster state: [] 06:56:30,685 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467: Set currentTerm=2, votedFor=null 06:56:30,685 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader465, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:30,685 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower466 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader465, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:30,685 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466: Set currentTerm=2, votedFor=null 06:56:30,686 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467 (Follower): In requestVote: RequestVote [term=2, candidateId=leader465, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:30,686 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466 (Follower): In requestVote: RequestVote [term=2, candidateId=leader465, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:30,686 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467: Set currentTerm=2, votedFor=leader465 06:56:30,686 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466: Set currentTerm=2, votedFor=leader465 06:56:30,686 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:30,686 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:30,687 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 06:56:30,687 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 06:56:30,687 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Election: Leader has following peers: [follower468, follower467, follower466] 06:56:30,687 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Checking sendAppendEntries for follower follower468: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:30,687 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Checking sendAppendEntries for follower follower467: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:30,688 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower468 (Follower): Term 2 in "AppendEntries [leaderId=leader465, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 06:56:30,688 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Checking sendAppendEntries for follower follower466: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:30,688 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower468: Set currentTerm=2, votedFor=null 06:56:30,688 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 06:56:30,688 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower467: Last sync leader does not match current leader leader465, need to catch up to -1 06:56:30,688 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower468: Last sync leader does not match current leader leader465, need to catch up to -1 06:56:30,688 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower467 updated: matchIndex: -1, nextIndex: 0 06:56:30,688 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower466: Last sync leader does not match current leader leader465, need to catch up to -1 06:56:30,688 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Checking sendAppendEntries for follower follower467: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:30,688 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower468 updated: matchIndex: -1, nextIndex: 0 [WARN] [01/22/2024 06:56:30.688] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower467] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3c69e726 06:56:30,688 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Checking sendAppendEntries for follower follower468: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:30.688] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower466] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@158ec28d 06:56:30,688 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower466 updated: matchIndex: -1, nextIndex: 0 06:56:30,688 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Checking sendAppendEntries for follower follower466: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:30.689] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower468] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@464b501b 06:56:30,712 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 06:56:30,712 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 06:56:30,713 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 06:56:30,713 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower467 starting 06:56:30,714 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower467: onShutDown 06:56:30,714 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower467 ending 06:56:30,714 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 06:56:30,714 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 06:56:30,714 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465: onRequestLeadership RequestLeadership [requestedFollowerId=follower467, replyTo=Actor[akka://test/user/$b#544413637]] 06:56:30,714 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465: Initiating leader transfer 06:56:30,715 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower468: onLeaderTransitioning: LeaderTransitioning [leaderId=leader465] 06:56:30,715 AM [test-akka.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader465: pauseLeader successfully completed - doing transfer 06:56:30,715 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Attempting to transfer leadership 06:56:30,715 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Checking sendAppendEntries for follower follower468: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:30.715] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower467] received dead letter from TestActor[akka://test/user/leader465]: LeaderTransitioning [leaderId=leader465] 06:56:30,715 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower466: onLeaderTransitioning: LeaderTransitioning [leaderId=leader465] 06:56:30,715 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Checking sendAppendEntries for follower follower467: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:30,715 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower468: Lagging 0 entries behind leader leader465 and reached -1 (of expected -1) 06:56:30,715 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Checking sendAppendEntries for follower follower466: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:30,715 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower466: Lagging 0 entries behind leader leader465 and reached -1 (of expected -1) [WARN] [01/22/2024 06:56:30.715] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower467] received dead letter from TestActor[akka://test/user/leader465]: AppendEntries [leaderId=leader465, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [01/22/2024 06:56:30.715] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower468] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@72198171 [WARN] [01/22/2024 06:56:30.715] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower466] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@50b1551d [WARN] [01/22/2024 06:56:30.805] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower467] received dead letter from TestActor[akka://test/user/leader465]: AppendEntries [leaderId=leader465, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 06:56:30,925 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower468: Lagging 0 entries behind leader leader465 and reached -1 (of expected -1) 06:56:30,925 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower466: Lagging 0 entries behind leader leader465 and reached -1 (of expected -1) [WARN] [01/22/2024 06:56:30.925] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower467] received dead letter from TestActor[akka://test/user/leader465]: AppendEntries [leaderId=leader465, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 06:56:31,44 AM [test-akka.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader465: leader transfer aborted 06:56:31,44 AM [test-akka.actor.default-dispatcher-11] [WARN] RaftActorLeadershipTransferCohort - leader465: Failed to transfer leadership in 329.7 ms 06:56:31,44 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465: LeadershipTransfer request from follower467 failed 06:56:31,45 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader465 (Leader): Leadership transfer expired 06:56:31,46 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower466: Lagging 0 entries behind leader leader465 and reached -1 (of expected -1) [WARN] [01/22/2024 06:56:31.045] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower467] received dead letter from TestActor[akka://test/user/leader465]: AppendEntries [leaderId=leader465, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 06:56:31,47 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower468: Lagging 0 entries behind leader leader465 and reached -1 (of expected -1) 06:56:31,68 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 06:56:31,68 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower466-notifier469#-1285379689] 06:56:31,79 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower466-collector470#-1689747349] 06:56:31,80 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower466] 06:56:31,80 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower467-notifier471#1355797991] 06:56:31,80 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower467-collector472#-142042624] 06:56:31,80 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower467] 06:56:31,81 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower468-notifier473#-1707522391] 06:56:31,81 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower468-collector474#1022198781] 06:56:31,81 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower468] 06:56:31,81 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader465-notifier475#-2076109975] 06:56:31,81 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader465-collector476#-1460831738] 06:56:31,82 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader465] 06:56:31,82 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$b#544413637] 06:56:31,83 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 06:56:31,83 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/22/2024 06:56:31.086] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower478]: org.opendaylight.controller.cluster.common.actor.Monitor@415a537 06:56:31,86 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower478 with journal batch size 1000 06:56:31,87 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: SnapshotOffer called. 06:56:31,87 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Moving last applied index from -1 to -1 06:56:31,87 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Set currentTerm=1, votedFor=null 06:56:31,87 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Updated server config: isVoting: true, peers: [PeerInfo [id=leader477, address=akka://test/user/leader477, votingState=VOTING], PeerInfo [id=follower480, address=akka://test/user/follower480, votingState=NON_VOTING], PeerInfo [id=follower479, address=akka://test/user/follower479, votingState=VOTING]] 06:56:31,88 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower478 in 72.95 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:31,88 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Recovery completed in 890.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:31.090] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower479]: org.opendaylight.controller.cluster.common.actor.Monitor@291eec91 06:56:31,90 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower479 with journal batch size 1000 06:56:31,91 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: SnapshotOffer called. 06:56:31,91 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Moving last applied index from -1 to -1 06:56:31,91 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Set currentTerm=1, votedFor=null 06:56:31,91 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Updated server config: isVoting: true, peers: [PeerInfo [id=leader477, address=akka://test/user/leader477, votingState=VOTING], PeerInfo [id=follower480, address=akka://test/user/follower480, votingState=NON_VOTING], PeerInfo [id=follower478, address=akka://test/user/follower478, votingState=VOTING]] 06:56:31,91 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower479 in 34.71 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:31,91 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Recovery completed in 456.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:31.193] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower480]: org.opendaylight.controller.cluster.common.actor.Monitor@1abdb78e 06:56:31,194 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower480 with journal batch size 1000 06:56:31,197 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: SnapshotOffer called. 06:56:31,197 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: Moving last applied index from -1 to -1 06:56:31,197 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: Set currentTerm=1, votedFor=null 06:56:31,197 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: Updated server config: isVoting: false, peers: [PeerInfo [id=leader477, address=akka://test/user/leader477, votingState=VOTING], PeerInfo [id=follower479, address=akka://test/user/follower479, votingState=VOTING], PeerInfo [id=follower478, address=akka://test/user/follower478, votingState=VOTING]] 06:56:31,198 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower480 in 80.83 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:31,198 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: Recovery completed in 985.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:31.295] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader477]: org.opendaylight.controller.cluster.common.actor.Monitor@7f3016f8 06:56:31,296 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader477 with journal batch size 1000 06:56:31,296 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: SnapshotOffer called. 06:56:31,296 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Moving last applied index from -1 to -1 06:56:31,296 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Set currentTerm=1, votedFor=null 06:56:31,296 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Updated server config: isVoting: true, peers: [PeerInfo [id=follower480, address=akka://test/user/follower480, votingState=NON_VOTING], PeerInfo [id=follower479, address=akka://test/user/follower479, votingState=VOTING], PeerInfo [id=follower478, address=akka://test/user/follower478, votingState=VOTING]] 06:56:31,297 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader477 in 87.73 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:31,298 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Recovery completed in 1.268 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 06:56:31,674 AM [test-akka.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 06:56:31,674 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.69:25520, Removed) 06:56:31,674 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Follower): Received ElectionTimeout - switching to Candidate 06:56:31,674 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Candidate): Election: Candidate has following voting peers: [follower479, follower478] 06:56:31,674 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Set currentTerm=2, votedFor=leader477 06:56:31,675 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Candidate): Starting new election term 2 06:56:31,675 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Candidate): Sending RequestVote [term=2, candidateId=leader477, lastLogIndex=-1, lastLogTerm=-1] to peer follower479 06:56:31,675 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Candidate): Sending RequestVote [term=2, candidateId=leader477, lastLogIndex=-1, lastLogTerm=-1] to peer follower478 06:56:31,675 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:31,675 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 06:56:31,675 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:31,675 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Cluster state: [] 06:56:31,675 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): Cluster state: [] 06:56:31,675 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader477, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:31,675 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader477, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:31,675 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader477, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:31,675 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader477, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:31,675 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Set currentTerm=2, votedFor=null 06:56:31,675 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Set currentTerm=2, votedFor=null 06:56:31,675 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): In requestVote: RequestVote [term=2, candidateId=leader477, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:31,675 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): In requestVote: RequestVote [term=2, candidateId=leader477, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:31,675 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Set currentTerm=2, votedFor=leader477 06:56:31,675 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Set currentTerm=2, votedFor=leader477 06:56:31,675 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:31,675 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:31,677 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 06:56:31,677 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 06:56:31,677 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Election: Leader has following peers: [follower480, follower479, follower478] 06:56:31,677 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower480: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:31,677 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower479: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:31,677 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower478: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:31,677 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 06:56:31,677 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): Term 2 in "AppendEntries [leaderId=leader477, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 06:56:31,677 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: Set currentTerm=2, votedFor=null 06:56:31,677 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower478: Last sync leader does not match current leader leader477, need to catch up to -1 06:56:31,677 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower480: Last sync leader does not match current leader leader477, need to catch up to -1 06:56:31,677 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower478 updated: matchIndex: -1, nextIndex: 0 06:56:31,677 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower478: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:31,677 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower480 updated: matchIndex: -1, nextIndex: 0 06:56:31,678 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower480: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:31,678 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower479: Last sync leader does not match current leader leader477, need to catch up to -1 06:56:31,678 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower479 updated: matchIndex: -1, nextIndex: 0 06:56:31,678 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower479: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:31.677] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower478] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6d815474 [WARN] [01/22/2024 06:56:31.678] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower479] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1ed033bf [WARN] [01/22/2024 06:56:31.679] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower480] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3c70d100 06:56:31,702 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 06:56:31,702 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower480: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): sendAppendEntries: 0 is present for follower follower480 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Sending AppendEntries to follower follower480: AppendEntries [leaderId=leader477, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower479: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): sendAppendEntries: 0 is present for follower follower479 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Sending AppendEntries to follower follower479: AppendEntries [leaderId=leader477, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower478: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): sendAppendEntries: 0 is present for follower follower478 06:56:31,702 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): handleAppendEntries: AppendEntries [leaderId=leader477, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:31,702 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Sending AppendEntries to follower follower478: AppendEntries [leaderId=leader477, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:31,702 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): Number of entries to be appended = 1 06:56:31,702 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 06:56:31,702 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): Append entry to log zero 06:56:31,702 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:31,703 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): Log size is now 1 06:56:31,703 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower480, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 06:56:31,703 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): handleAppendEntries: AppendEntries [leaderId=leader477, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:31,703 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower480: Lagging 0 entries behind leader leader477 and reached -1 (of expected -1) 06:56:31,703 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): Number of entries to be appended = 1 06:56:31,703 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 06:56:31,703 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): Append entry to log zero 06:56:31,703 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:31,703 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): Log size is now 1 06:56:31,703 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower480 updated: matchIndex: 0, nextIndex: 1 06:56:31,703 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower478, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 06:56:31,703 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower478: Lagging 0 entries behind leader leader477 and reached -1 (of expected -1) 06:56:31,703 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower480: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,703 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower478 updated: matchIndex: 0, nextIndex: 1 06:56:31,703 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower478: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,704 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:31,704 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:31,704 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 06:56:31,704 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Setting last applied to 0 06:56:31,704 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Moving last applied index from -1 to 0 06:56:31,704 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Applying state for log index 0 data zero [WARN] [01/22/2024 06:56:31.704] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower480] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6df3b794 06:56:31,704 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:31,704 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: applyState called: zero 06:56:31,704 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Persisting ApplyJournalEntries with index=0 [WARN] [01/22/2024 06:56:31.704] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower478] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@e8a8a0d 06:56:31,794 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): sendAppendEntries: 0 is present for follower follower479 06:56:31,794 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Sending AppendEntries to follower follower479: AppendEntries [leaderId=leader477, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:31,915 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): sendAppendEntries: 0 is present for follower follower479 06:56:31,915 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Sending AppendEntries to follower follower479: AppendEntries [leaderId=leader477, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:31,915 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): Commit index set to 0 06:56:31,915 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower480: Lagging 0 entries behind leader leader477 and reached 0 (of expected -1) 06:56:31,915 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 06:56:31,915 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): Setting last applied to 0 06:56:31,915 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: Moving last applied index from -1 to 0 06:56:31,915 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: Applying state for log index 0 data zero 06:56:31,915 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: applyState called: zero 06:56:31,915 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: Persisting ApplyJournalEntries with index=0 06:56:31,916 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): Commit index set to 0 06:56:31,916 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower478: Lagging 0 entries behind leader leader477 and reached 0 (of expected -1) 06:56:31,916 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 06:56:31,916 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): Setting last applied to 0 06:56:31,916 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Moving last applied index from -1 to 0 06:56:31,916 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Applying state for log index 0 data zero 06:56:31,916 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: applyState called: zero 06:56:31,916 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Persisting ApplyJournalEntries with index=0 06:56:31,954 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 06:56:31,954 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 06:56:31,955 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: onShutDown 06:56:31,955 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Initiating leader transfer 06:56:31,955 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: onLeaderTransitioning: LeaderTransitioning [leaderId=leader477] 06:56:31,955 AM [test-akka.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader477: pauseLeader successfully completed - doing transfer 06:56:31,955 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Attempting to transfer leadership 06:56:31,955 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower480: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,955 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: onLeaderTransitioning: LeaderTransitioning [leaderId=leader477] 06:56:31,955 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower479: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,955 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): sendAppendEntries: 0 is present for follower follower479 06:56:31,955 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower478: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,955 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower480: Lagging 0 entries behind leader leader477 and reached 0 (of expected -1) 06:56:31,955 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower478: Lagging 0 entries behind leader leader477 and reached 0 (of expected -1) 06:56:31,955 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): tryToCompleteLeadershipTransfer: followerId: follower480, matchIndex: 0, lastIndex: 0, isVoting: false 06:56:31,956 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): tryToCompleteLeadershipTransfer: followerId: follower478, matchIndex: 0, lastIndex: 0, isVoting: true 06:56:31,956 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Follower's log matches - sending ElectionTimeout 06:56:31,956 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower480: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,956 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower479: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,956 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): sendAppendEntries: 0 is present for follower follower479 06:56:31,956 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Checking sendAppendEntries for follower follower478: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,956 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Leader transfer complete 06:56:31,956 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower480: Lagging 0 entries behind leader leader477 and reached 0 (of expected -1) 06:56:31,956 AM [test-akka.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader477: leader transfer complete - waiting for new leader 06:56:31,956 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower478: Lagging 0 entries behind leader leader477 and reached 0 (of expected -1) 06:56:31,956 AM [test-akka.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 06:56:31,956 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower): Received TimeoutNow - switching to Candidate 06:56:31,956 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Candidate): Election: Candidate has following voting peers: [leader477, follower479] 06:56:31,956 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478: Set currentTerm=3, votedFor=follower478 06:56:31,956 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Candidate): Starting new election term 3 06:56:31,956 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Candidate): Sending RequestVote [term=3, candidateId=follower478, lastLogIndex=0, lastLogTerm=2] to peer leader477 06:56:31,956 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Candidate): Sending RequestVote [term=3, candidateId=follower478, lastLogIndex=0, lastLogTerm=2] to peer follower479 06:56:31,956 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 06:56:31,957 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:31,957 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:31,957 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Cluster state: [] 06:56:31,957 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Cluster state: [] 06:56:31,957 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower478, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 06:56:31,957 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower478, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 06:56:31,957 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower478, lastLogIndex=0, lastLogTerm=2]" message is greater than follower's term 2 - updating term 06:56:31,957 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Set currentTerm=3, votedFor=null 06:56:31,957 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower478, lastLogIndex=0, lastLogTerm=2]" message is greater than leader's term 2 - switching to Follower 06:56:31,957 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Set currentTerm=3, votedFor=null 06:56:31,957 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): In requestVote: RequestVote [term=3, candidateId=follower478, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:31,957 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Set currentTerm=3, votedFor=follower478 06:56:31,957 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): Leadership transfer in progress - processing RequestVote 06:56:31,957 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 06:56:31,957 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): In requestVote: RequestVote [term=3, candidateId=follower478, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 06:56:31,957 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: Set currentTerm=3, votedFor=follower478 06:56:31,957 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 06:56:31,957 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader477 (Leader) :- Switching from behavior Leader to Follower, election term: 3 06:56:31,958 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 06:56:31,958 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 06:56:31,958 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): Election: Leader has following peers: [leader477, follower480, follower479] 06:56:31,958 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): Checking sendAppendEntries for follower leader477: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,958 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): Checking sendAppendEntries for follower follower480: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,958 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): Checking sendAppendEntries for follower follower479: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,958 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - leader477: Last sync leader does not match current leader follower478, need to catch up to -1 06:56:31,958 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 06:56:31,958 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader477: leader changed to follower478 06:56:31,958 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower480 (Follower): Term 3 in "AppendEntries [leaderId=follower478, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 06:56:31,958 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower480: Set currentTerm=3, votedFor=null 06:56:31,959 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower480: Last sync leader does not match current leader follower478, need to catch up to -1 06:56:31,959 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader477 updated: matchIndex: 0, nextIndex: 1 06:56:31,959 AM [test-akka.actor.default-dispatcher-5] [INFO] RaftActorLeadershipTransferCohort - leader477: Successfully transferred leadership to follower478 in 3.642 ms 06:56:31,959 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader477: leader transfer succeeded - sending PoisonPill 06:56:31,959 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): Checking sendAppendEntries for follower leader477: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:31,959 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - leader477: Lagging 0 entries behind leader follower478 and reached 0 (of expected -1) 06:56:31,959 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower480 updated: matchIndex: 0, nextIndex: 1 06:56:31,959 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): Checking sendAppendEntries for follower follower480: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:31.959] [test-akka.actor.default-dispatcher-18] [akka://test/user/leader477] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7410a138 [WARN] [01/22/2024 06:56:31.959] [test-akka.actor.default-dispatcher-11] [akka://test/user/leader477] received dead letter: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7324044b 06:56:31,960 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower480: Lagging 0 entries behind leader follower478 and reached 0 (of expected -1) [WARN] [01/22/2024 06:56:31.960] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower480] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@589b0084 [WARN] [01/22/2024 06:56:31.960] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower480] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@66b700ef 06:56:32,475 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower480: Lagging 0 entries behind leader follower478 and reached 0 (of expected -1) [WARN] [01/22/2024 06:56:32.475] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader477] received dead letter from TestActor[akka://test/user/follower478]: AppendEntries [leaderId=follower478, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 06:56:32,476 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower479: Last sync leader does not match current leader follower478, need to catch up to -1 06:56:32,476 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): sendAppendEntries: 0 is present for follower follower479 06:56:32,476 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): Sending AppendEntries to follower follower479: AppendEntries [leaderId=follower478, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] [WARN] [01/22/2024 06:56:32.476] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower479] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@10b217f7 06:56:32,476 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): handleAppendEntries: AppendEntries [leaderId=follower478, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:32,476 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Number of entries to be appended = 1 06:56:32,476 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 06:56:32,476 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Append entry to log zero 06:56:32,476 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:32,476 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Log size is now 1 06:56:32,477 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Commit index set to 0 06:56:32,477 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): handleAppendEntries returning : AppendEntriesReply [term=3, success=true, followerId=follower479, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 06:56:32,477 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower479: Lagging 0 entries behind leader follower478 and reached 0 (of expected -1) 06:56:32,477 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 06:56:32,477 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower479 updated: matchIndex: 0, nextIndex: 1 06:56:32,477 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479 (Follower): Setting last applied to 0 06:56:32,477 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Moving last applied index from -1 to 0 06:56:32,477 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Applying state for log index 0 data zero 06:56:32,477 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower478 (Leader): Checking sendAppendEntries for follower follower479: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:32,477 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: applyState called: zero 06:56:32,478 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:32,478 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: Persisting ApplyJournalEntries with index=0 [WARN] [01/22/2024 06:56:32.479] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower479] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@65ef366 06:56:32,510 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: onLeaderTransitioning: LeaderTransitioning [leaderId=leader477] 06:56:32,511 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 06:56:32,511 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower479 starting 06:56:32,511 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower479: onShutDown 06:56:32,512 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower479 ending 06:56:32,512 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 06:56:32,512 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower478-notifier481#-2144871385] 06:56:32,523 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower478-collector482#796851854] 06:56:32,523 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower478] 06:56:32,523 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower479-notifier483#2027310215] 06:56:32,524 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower479-collector484#1608288077] 06:56:32,524 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower479] 06:56:32,524 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower480-notifier485#-479533286] 06:56:32,524 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower480-collector486#1015003432] 06:56:32,524 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower480] 06:56:32,525 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader477-notifier487#159838678] 06:56:32,525 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader477-collector488#-1914428552] 06:56:32,525 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader477] 06:56:32,526 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [01/22/2024 06:56:32.529] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader489]: org.opendaylight.controller.cluster.common.actor.Monitor@31e7f16b 06:56:32,529 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader489 with journal batch size 1000 06:56:32,531 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader489: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 06:56:32,531 AM [test-akka.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 06:56:32,531 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader489 (Follower): Received TimeoutNow - switching to Candidate 06:56:32,531 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader489 (Candidate): Election: Candidate has following voting peers: [] 06:56:32,531 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader489: Set currentTerm=1, votedFor=leader489 06:56:32,532 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader489 (Candidate): Starting new election term 1 06:56:32,532 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader489 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 06:56:32,532 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader489 (Candidate): Received ElectionTimeout 06:56:32,532 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader489 (Leader): Election: Leader has following peers: [] 06:56:32,532 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader489 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 06:56:32,628 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader489 starting 06:56:32,629 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader489: onShutDown 06:56:32,630 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader489 ending 06:56:32,630 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 06:56:32,630 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader489-collector493#-2078467970] 06:56:32,641 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader489] 06:56:32,642 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 06:56:32,642 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 06:56:32,645 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower495 with journal batch size 1000 [WARN] [01/22/2024 06:56:32.645] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower495]: org.opendaylight.controller.cluster.common.actor.Monitor@64290327 06:56:32,646 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: SnapshotOffer called. 06:56:32,646 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: Moving last applied index from -1 to -1 06:56:32,646 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: Set currentTerm=1, votedFor=null 06:56:32,646 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: Updated server config: isVoting: true, peers: [PeerInfo [id=follower497, address=akka://test/user/follower497, votingState=NON_VOTING], PeerInfo [id=leader494, address=akka://test/user/leader494, votingState=VOTING], PeerInfo [id=follower496, address=akka://test/user/follower496, votingState=VOTING]] 06:56:32,646 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower495 in 134.4 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:32,647 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: Recovery completed in 1.494 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:32.746] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower496]: org.opendaylight.controller.cluster.common.actor.Monitor@cb6589b 06:56:32,747 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower496 with journal batch size 1000 06:56:32,747 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496: SnapshotOffer called. 06:56:32,747 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496: Moving last applied index from -1 to -1 06:56:32,747 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496: Set currentTerm=1, votedFor=null 06:56:32,747 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496: Updated server config: isVoting: true, peers: [PeerInfo [id=follower497, address=akka://test/user/follower497, votingState=NON_VOTING], PeerInfo [id=leader494, address=akka://test/user/leader494, votingState=VOTING], PeerInfo [id=follower495, address=akka://test/user/follower495, votingState=VOTING]] 06:56:32,748 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower496 in 125.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:32,749 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower496: Recovery completed in 1.242 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:32.848] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower497]: org.opendaylight.controller.cluster.common.actor.Monitor@22e2d66a 06:56:32,848 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower497 with journal batch size 1000 06:56:32,849 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: SnapshotOffer called. 06:56:32,849 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: Moving last applied index from -1 to -1 06:56:32,849 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: Set currentTerm=1, votedFor=null 06:56:32,849 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: Updated server config: isVoting: false, peers: [PeerInfo [id=leader494, address=akka://test/user/leader494, votingState=VOTING], PeerInfo [id=follower496, address=akka://test/user/follower496, votingState=VOTING], PeerInfo [id=follower495, address=akka://test/user/follower495, votingState=VOTING]] 06:56:32,849 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower497 in 73.88 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:32,851 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: Recovery completed in 1.924 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:32.950] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader494]: org.opendaylight.controller.cluster.common.actor.Monitor@2db1aefa 06:56:32,951 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader494 with journal batch size 1000 06:56:32,951 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: SnapshotOffer called. 06:56:32,951 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Moving last applied index from -1 to -1 06:56:32,951 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Set currentTerm=1, votedFor=null 06:56:32,951 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Updated server config: isVoting: true, peers: [PeerInfo [id=follower497, address=akka://test/user/follower497, votingState=NON_VOTING], PeerInfo [id=follower496, address=akka://test/user/follower496, votingState=VOTING], PeerInfo [id=follower495, address=akka://test/user/follower495, votingState=VOTING]] 06:56:32,952 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader494 in 78.69 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:32,953 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Recovery completed in 1.608 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 06:56:33,304 AM [test-akka.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 06:56:33,304 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.69:25520, Removed) 06:56:33,304 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Follower): Received ElectionTimeout - switching to Candidate 06:56:33,304 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Candidate): Election: Candidate has following voting peers: [follower496, follower495] 06:56:33,304 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Set currentTerm=2, votedFor=leader494 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Candidate): Starting new election term 2 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Candidate): Sending RequestVote [term=2, candidateId=leader494, lastLogIndex=-1, lastLogTerm=-1] to peer follower496 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Candidate): Sending RequestVote [term=2, candidateId=leader494, lastLogIndex=-1, lastLogTerm=-1] to peer follower495 06:56:33,305 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 06:56:33,305 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496 (Follower): Cluster state: [] 06:56:33,305 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader494, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:33,305 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower496 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader494, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:33,305 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496: Set currentTerm=2, votedFor=null 06:56:33,305 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496 (Follower): In requestVote: RequestVote [term=2, candidateId=leader494, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:33,305 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496: Set currentTerm=2, votedFor=leader494 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:33,305 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): Cluster state: [] 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader494, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader494, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: Set currentTerm=2, votedFor=null 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): In requestVote: RequestVote [term=2, candidateId=leader494, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: Set currentTerm=2, votedFor=leader494 06:56:33,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:33,306 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 06:56:33,306 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 06:56:33,307 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Election: Leader has following peers: [follower497, follower496, follower495] 06:56:33,307 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower497: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:33,307 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower496: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:33,307 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower497 (Follower): Term 2 in "AppendEntries [leaderId=leader494, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 06:56:33,307 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower495: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:33,307 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: Set currentTerm=2, votedFor=null 06:56:33,307 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 06:56:33,307 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower497: Last sync leader does not match current leader leader494, need to catch up to -1 06:56:33,307 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower497 updated: matchIndex: -1, nextIndex: 0 06:56:33,307 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower497: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:33,307 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower496: Last sync leader does not match current leader leader494, need to catch up to -1 06:56:33,307 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower495: Last sync leader does not match current leader leader494, need to catch up to -1 06:56:33,308 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower496 updated: matchIndex: -1, nextIndex: 0 06:56:33,308 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower496: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:33,308 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower495 updated: matchIndex: -1, nextIndex: 0 06:56:33,308 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower495: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:33.308] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower496] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@69949ef8 [WARN] [01/22/2024 06:56:33.308] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower495] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6317adeb [WARN] [01/22/2024 06:56:33.309] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower497] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@13fd5957 06:56:33,352 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 06:56:33,352 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 06:56:33,352 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 06:56:33,352 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 06:56:33,352 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower497: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): sendAppendEntries: 0 is present for follower follower497 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Sending AppendEntries to follower follower497: AppendEntries [leaderId=leader494, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower496: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): sendAppendEntries: 0 is present for follower follower496 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Sending AppendEntries to follower follower496: AppendEntries [leaderId=leader494, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower495: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): sendAppendEntries: 0 is present for follower follower495 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Sending AppendEntries to follower follower495: AppendEntries [leaderId=leader494, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:33,353 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497 (Follower): handleAppendEntries: AppendEntries [leaderId=leader494, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:33,353 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497 (Follower): Number of entries to be appended = 1 06:56:33,353 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 06:56:33,353 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497 (Follower): Append entry to log zero 06:56:33,353 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): handleAppendEntries: AppendEntries [leaderId=leader494, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): Number of entries to be appended = 1 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): Append entry to log zero 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:33,353 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497 (Follower): Log size is now 1 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): Log size is now 1 06:56:33,353 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower497, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower495, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 06:56:33,353 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower497: Lagging 0 entries behind leader leader494 and reached -1 (of expected -1) 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower495: Lagging 0 entries behind leader leader494 and reached -1 (of expected -1) 06:56:33,353 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower497 updated: matchIndex: 0, nextIndex: 1 06:56:33,354 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower497: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:33,354 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower495 updated: matchIndex: 0, nextIndex: 1 06:56:33,354 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower495: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:33,354 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 06:56:33,354 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 06:56:33,354 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Setting last applied to 0 06:56:33,354 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Moving last applied index from -1 to 0 06:56:33,354 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Applying state for log index 0 data zero 06:56:33,354 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: applyState called: zero 06:56:33,354 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Persisting ApplyJournalEntries with index=0 06:56:33,354 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [01/22/2024 06:56:33.354] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower497] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@648a2e0f 06:56:33,354 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [01/22/2024 06:56:33.355] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower495] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@44ec9a1a 06:56:33,424 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): sendAppendEntries: 0 is present for follower follower496 06:56:33,425 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Sending AppendEntries to follower follower496: AppendEntries [leaderId=leader494, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:33,545 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): sendAppendEntries: 0 is present for follower follower496 06:56:33,545 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Sending AppendEntries to follower follower496: AppendEntries [leaderId=leader494, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 06:56:33,545 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497 (Follower): Commit index set to 0 06:56:33,545 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower497: Lagging 0 entries behind leader leader494 and reached 0 (of expected -1) 06:56:33,545 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 06:56:33,545 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497 (Follower): Setting last applied to 0 06:56:33,545 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: Moving last applied index from -1 to 0 06:56:33,545 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: Applying state for log index 0 data zero 06:56:33,545 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: applyState called: zero 06:56:33,545 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): Commit index set to 0 06:56:33,545 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower495: Lagging 0 entries behind leader leader494 and reached 0 (of expected -1) 06:56:33,545 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 06:56:33,545 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495 (Follower): Setting last applied to 0 06:56:33,545 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: Persisting ApplyJournalEntries with index=0 06:56:33,545 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: Moving last applied index from -1 to 0 06:56:33,545 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: Applying state for log index 0 data zero 06:56:33,545 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: applyState called: zero 06:56:33,546 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: Persisting ApplyJournalEntries with index=0 06:56:33,555 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 06:56:33,555 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 06:56:33,555 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 06:56:33,555 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: onRequestLeadership RequestLeadership [requestedFollowerId=follower496, replyTo=Actor[akka://test/user/$c#1915830658]] 06:56:33,555 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: Initiating leader transfer 06:56:33,555 AM [test-akka.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader494: pauseLeader successfully completed - doing transfer 06:56:33,555 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Attempting to transfer leadership 06:56:33,555 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower497: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:33,555 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower496: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:33,555 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): sendAppendEntries: 0 is present for follower follower496 06:56:33,555 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Checking sendAppendEntries for follower follower495: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 06:56:33,555 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower495: onLeaderTransitioning: LeaderTransitioning [leaderId=leader494] 06:56:33,555 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower495: Lagging 0 entries behind leader leader494 and reached 0 (of expected -1) 06:56:33,555 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower497: Lagging 0 entries behind leader leader494 and reached 0 (of expected -1) 06:56:33,555 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower497: onLeaderTransitioning: LeaderTransitioning [leaderId=leader494] 06:56:33,556 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower496: onLeaderTransitioning: LeaderTransitioning [leaderId=leader494] 06:56:33,664 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower497: Lagging 0 entries behind leader leader494 and reached 0 (of expected -1) 06:56:33,664 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower495: Lagging 0 entries behind leader leader494 and reached 0 (of expected -1) 06:56:33,784 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower495: Lagging 0 entries behind leader leader494 and reached 0 (of expected -1) 06:56:33,784 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower497: Lagging 0 entries behind leader leader494 and reached 0 (of expected -1) 06:56:33,905 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader494: leader transfer aborted 06:56:33,905 AM [test-akka.actor.default-dispatcher-17] [WARN] RaftActorLeadershipTransferCohort - leader494: Failed to transfer leadership in 349.9 ms 06:56:33,905 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494: LeadershipTransfer request from follower496 failed 06:56:33,905 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader494 (Leader): Leadership transfer expired 06:56:33,906 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower497: Lagging 0 entries behind leader leader494 and reached 0 (of expected -1) 06:56:33,906 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower495: Lagging 0 entries behind leader leader494 and reached 0 (of expected -1) 06:56:33,908 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 06:56:33,908 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower495-notifier498#669196137] 06:56:33,920 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower495-collector499#-2101824418] 06:56:33,920 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower495] 06:56:33,920 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower496-notifier500#2035489508] 06:56:33,921 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower496-collector501#-733500264] 06:56:33,921 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower496] 06:56:33,922 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower497-notifier502#1485550799] 06:56:33,922 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower497-collector503#-767679345] 06:56:33,922 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower497] 06:56:33,923 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader494-notifier504#-2124568966] 06:56:33,923 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader494-collector505#-1033769537] 06:56:33,923 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader494] 06:56:33,923 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$c#1915830658] 06:56:33,924 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 06:56:33,924 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/22/2024 06:56:33.926] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower507]: org.opendaylight.controller.cluster.common.actor.Monitor@1decc411 06:56:33,927 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower507 with journal batch size 1000 06:56:33,930 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: SnapshotOffer called. 06:56:33,930 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Moving last applied index from -1 to -1 06:56:33,930 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Set currentTerm=1, votedFor=null 06:56:33,930 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Updated server config: isVoting: true, peers: [PeerInfo [id=follower508, address=akka://test/user/follower508, votingState=VOTING], PeerInfo [id=leader506, address=akka://test/user/leader506, votingState=VOTING], PeerInfo [id=follower509, address=akka://test/user/follower509, votingState=NON_VOTING]] 06:56:33,930 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower507 in 71.60 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:33,931 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Recovery completed in 1.158 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:33.932] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower508]: org.opendaylight.controller.cluster.common.actor.Monitor@5c24c2e4 06:56:33,932 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower508 with journal batch size 1000 06:56:33,932 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: SnapshotOffer called. 06:56:33,932 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Moving last applied index from -1 to -1 06:56:33,932 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Set currentTerm=1, votedFor=null 06:56:33,932 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Updated server config: isVoting: true, peers: [PeerInfo [id=follower507, address=akka://test/user/follower507, votingState=VOTING], PeerInfo [id=leader506, address=akka://test/user/leader506, votingState=VOTING], PeerInfo [id=follower509, address=akka://test/user/follower509, votingState=NON_VOTING]] 06:56:33,932 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower508 in 64.44 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:33,933 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Recovery completed in 808.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:34.033] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower509]: org.opendaylight.controller.cluster.common.actor.Monitor@1a66ad6b 06:56:34,34 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower509 with journal batch size 1000 06:56:34,37 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: SnapshotOffer called. 06:56:34,37 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: Moving last applied index from -1 to -1 06:56:34,37 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: Set currentTerm=1, votedFor=null 06:56:34,37 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: Updated server config: isVoting: false, peers: [PeerInfo [id=follower508, address=akka://test/user/follower508, votingState=VOTING], PeerInfo [id=follower507, address=akka://test/user/follower507, votingState=VOTING], PeerInfo [id=leader506, address=akka://test/user/leader506, votingState=VOTING]] 06:56:34,37 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower509 in 87.95 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:34,38 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: Recovery completed in 1.196 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/22/2024 06:56:34.136] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader506]: org.opendaylight.controller.cluster.common.actor.Monitor@4ff78d21 06:56:34,137 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader506 with journal batch size 1000 06:56:34,137 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: SnapshotOffer called. 06:56:34,137 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Moving last applied index from -1 to -1 06:56:34,137 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Set currentTerm=1, votedFor=null 06:56:34,137 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Updated server config: isVoting: true, peers: [PeerInfo [id=follower508, address=akka://test/user/follower508, votingState=VOTING], PeerInfo [id=follower507, address=akka://test/user/follower507, votingState=VOTING], PeerInfo [id=follower509, address=akka://test/user/follower509, votingState=NON_VOTING]] 06:56:34,138 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader506 in 98.58 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 06:56:34,139 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Recovery completed in 1.544 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 06:56:34,464 AM [test-akka.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 06:56:34,464 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.69:25520, Removed) 06:56:34,464 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Follower): Received ElectionTimeout - switching to Candidate 06:56:34,464 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): Election: Candidate has following voting peers: [follower508, follower507] 06:56:34,464 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Set currentTerm=2, votedFor=leader506 06:56:34,464 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): Starting new election term 2 06:56:34,464 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): Sending RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] to peer follower508 06:56:34,464 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): Sending RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] to peer follower507 06:56:34,464 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:34,465 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 06:56:34,465 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): Cluster state: [] 06:56:34,465 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:34,465 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:34,465 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Set currentTerm=2, votedFor=null 06:56:34,465 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): In requestVote: RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:34,465 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:34,465 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Set currentTerm=2, votedFor=leader506 06:56:34,465 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): Cluster state: [] 06:56:34,465 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:34,465 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:34,465 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 06:56:34,465 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Set currentTerm=2, votedFor=null 06:56:34,465 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): In requestVote: RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:34,465 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Set currentTerm=2, votedFor=leader506 06:56:34,465 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 06:56:34,467 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 06:56:34,467 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 06:56:34,467 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Election: Leader has following peers: [follower508, follower507, follower509] 06:56:34,467 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower508: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,467 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower507: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,467 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower509: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,467 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower508: Last sync leader does not match current leader leader506, need to catch up to -1 06:56:34,467 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 06:56:34,467 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower507: Last sync leader does not match current leader leader506, need to catch up to -1 06:56:34,468 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower508 updated: matchIndex: -1, nextIndex: 0 06:56:34,468 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower508: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:34.467] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower507] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@69cf4d5c 06:56:34,468 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower509 (Follower): Term 2 in "AppendEntries [leaderId=leader506, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term [WARN] [01/22/2024 06:56:34.468] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower508] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@761dd611 06:56:34,468 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower507 updated: matchIndex: -1, nextIndex: 0 06:56:34,468 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: Set currentTerm=2, votedFor=null 06:56:34,468 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower507: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,468 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower509: Last sync leader does not match current leader leader506, need to catch up to -1 06:56:34,468 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower509 updated: matchIndex: -1, nextIndex: 0 06:56:34,468 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower509: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:34.469] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower509] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@14e7f1b8 06:56:34,487 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 06:56:34,487 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 06:56:34,488 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 06:56:34,488 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower507] 06:56:34,499 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower509] 06:56:34,510 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 06:56:34,510 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 06:56:34,510 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: onRequestLeadership RequestLeadership [requestedFollowerId=follower508, replyTo=Actor[akka://test/user/$d#519058574]] 06:56:34,510 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Initiating leader transfer 06:56:34,510 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: onLeaderTransitioning: LeaderTransitioning [leaderId=leader506] 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader506: pauseLeader successfully completed - doing transfer 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Attempting to transfer leadership [WARN] [01/22/2024 06:56:34.510] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower507] received dead letter from TestActor[akka://test/user/leader506]: LeaderTransitioning [leaderId=leader506] 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower508: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:34.511] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower509] received dead letter from TestActor[akka://test/user/leader506]: LeaderTransitioning [leaderId=leader506] 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower507: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,511 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower508: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower509: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): tryToCompleteLeadershipTransfer: followerId: follower508, matchIndex: -1, lastIndex: -1, isVoting: true 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Follower's log matches - sending ElectionTimeout 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower508: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/22/2024 06:56:34.511] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower507] received dead letter from TestActor[akka://test/user/leader506]: AppendEntries [leaderId=leader506, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower507: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,511 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower508: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower509: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Leader transfer complete 06:56:34,511 AM [test-akka.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader506: leader transfer complete - waiting for new leader 06:56:34,512 AM [test-akka.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 06:56:34,512 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): Received TimeoutNow - switching to Candidate 06:56:34,512 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Candidate): Election: Candidate has following voting peers: [follower507, leader506] 06:56:34,512 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Set currentTerm=3, votedFor=follower508 06:56:34,512 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Candidate): Starting new election term 3 06:56:34,512 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Candidate): Sending RequestVote [term=3, candidateId=follower508, lastLogIndex=-1, lastLogTerm=-1] to peer follower507 06:56:34,512 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Candidate): Sending RequestVote [term=3, candidateId=follower508, lastLogIndex=-1, lastLogTerm=-1] to peer leader506 06:56:34,512 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 06:56:34,512 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 06:56:34,512 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Cluster state: [] 06:56:34,512 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower508, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 06:56:34,512 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower508, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 06:56:34,512 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Set currentTerm=3, votedFor=null 06:56:34,512 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Leadership transfer in progress - processing RequestVote 06:56:34,512 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): In requestVote: RequestVote [term=3, candidateId=follower508, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 06:56:34,513 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Set currentTerm=3, votedFor=follower508 06:56:34,513 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 06:56:34,513 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader) :- Switching from behavior Leader to Follower, election term: 3 [WARN] [01/22/2024 06:56:34.513] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower509] received dead letter from TestActor[akka://test/user/leader506]: AppendEntries [leaderId=leader506, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [01/22/2024 06:56:34.513] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower508] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6f6bf646 [WARN] [01/22/2024 06:56:34.513] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower507] received dead letter from TestActor[akka://test/user/leader506]: AppendEntries [leaderId=leader506, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [01/22/2024 06:56:34.513] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower509] received dead letter from TestActor[akka://test/user/leader506]: AppendEntries [leaderId=leader506, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [01/22/2024 06:56:34.514] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower507] received dead letter from TestActor[akka://test/user/follower508]: RequestVote [term=3, candidateId=follower508, lastLogIndex=-1, lastLogTerm=-1] 06:56:34,514 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 06:56:34,515 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 06:56:34,515 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Leader): Election: Leader has following peers: [follower507, leader506, follower509] 06:56:34,515 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Leader): Checking sendAppendEntries for follower follower507: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,515 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Leader): Checking sendAppendEntries for follower leader506: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,515 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Leader): Checking sendAppendEntries for follower follower509: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,516 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [01/22/2024 06:56:34.516] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower507] received dead letter from TestActor[akka://test/user/follower508]: AppendEntries [leaderId=follower508, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [01/22/2024 06:56:34.516] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower509] received dead letter from TestActor[akka://test/user/follower508]: AppendEntries [leaderId=follower508, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 06:56:34,516 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - leader506: Last sync leader does not match current leader follower508, need to catch up to -1 06:56:34,516 AM [test-akka.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader506: leader changed to follower508 06:56:34,517 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader506 updated: matchIndex: -1, nextIndex: 0 06:56:34,517 AM [test-akka.actor.default-dispatcher-12] [INFO] RaftActorLeadershipTransferCohort - leader506: Successfully transferred leadership to follower508 in 6.047 ms 06:56:34,517 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Leader): Checking sendAppendEntries for follower leader506: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 06:56:34,517 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Leadership transferred successfully to follower508 [WARN] [01/22/2024 06:56:34.517] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader506] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@50dbb280 06:56:34,562 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 06:56:34,562 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower507-notifier510#-1014036103] 06:56:34,572 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower507-collector511#-663212470] 06:56:34,573 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower508-notifier512#1390942174] 06:56:34,573 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower508-collector513#1521718608] 06:56:34,573 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower508] 06:56:34,574 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower509-notifier514#-1084375063] 06:56:34,574 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower509-collector515#2076746208] 06:56:34,574 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader506-notifier516#-500309267] 06:56:34,574 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader506-collector517#-887553853] 06:56:34,575 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader506] 06:56:34,575 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$d#519058574] 06:56:34,576 AM [test-akka.actor.default-dispatcher-6] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [INFO] [01/22/2024 06:56:34.576] [main] [CoordinatedShutdown(akka://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [01/22/2024 06:56:34.577] [test-akka.actor.default-dispatcher-11] [akka://test/system/cluster/core/daemon] unhandled message from Actor[akka://test/deadLetters]: Leave(akka://test@10.30.171.69:25520) [INFO] [01/22/2024 06:56:34.578] [test-akka.actor.internal-dispatcher-10] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.69:25520] - Exiting completed [INFO] [01/22/2024 06:56:34.578] [test-akka.actor.internal-dispatcher-10] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.69:25520] - Shutting down... [INFO] [01/22/2024 06:56:34.579] [test-akka.actor.internal-dispatcher-10] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.69:25520] - Successfully shut down 06:56:34,581 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.69:25520] - Exiting completed 06:56:34,581 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.69:25520] - Shutting down... 06:56:34,581 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.69:25520] - Successfully shut down [INFO] [01/22/2024 06:56:34.581] [test-akka.remote.default-remote-dispatcher-7] [akka://test@10.30.171.69:25520/system/remoting-terminator] Shutting down remote daemon. 06:56:34,581 AM [test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 06:56:34,581 AM [test-akka.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [01/22/2024 06:56:34.581] [test-akka.remote.default-remote-dispatcher-7] [akka://test@10.30.171.69:25520/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. [INFO] [01/22/2024 06:56:34.586] [test-akka.remote.default-remote-dispatcher-7] [akka://test@10.30.171.69:25520/system/remoting-terminator] Remoting shut down. 06:56:34,586 AM [test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [DEBUG] [01/22/2024 06:56:34.586] [test-akka.actor.internal-dispatcher-4] [EventStream] shutting down: StandardOutLogger