[DEBUG] [10/29/2024 01:13:38.528] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 01:13:38,531 AM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [10/29/2024 01:13:38.531] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [10/29/2024 01:13:38.531] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [10/29/2024 01:13:38.562] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.219:17355] with UID [-1184904128468337445] 01:13:38,563 AM [test-pekko.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.219:17355] with UID [-1184904128468337445] [INFO] [10/29/2024 01:13:38.563] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.219:17355] - Starting up, Pekko version [1.0.3] ... 01:13:38,564 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.219:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [10/29/2024 01:13:38.566] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.219:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 01:13:38,566 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.219:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [10/29/2024 01:13:38.566] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.219:17355] - Started up successfully 01:13:38,566 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.219:17355] - Started up successfully [INFO] [10/29/2024 01:13:38.567] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.219:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [10/29/2024 01:13:38.567] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.219:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 01:13:38,567 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.219:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 01:13:38,567 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.219:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 01:13:38,584 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 01:13:38,584 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:13:38,591 AM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower551 with journal batch size 1000 01:13:38,591 AM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower551] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [10/29/2024 01:13:38.592] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower551]: org.opendaylight.controller.cluster.common.actor.Monitor@432cf48e [INFO] [pekkoDeadLetter][10/29/2024 01:13:38.591] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower551] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:38,594 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: SnapshotOffer called. 01:13:38,594 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Moving last applied index from -1 to -1 01:13:38,594 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=1, votedFor=null 01:13:38,594 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING]] 01:13:38,594 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower551 in 114.6 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [10/29/2024 01:13:38.596] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower552]: org.opendaylight.controller.cluster.common.actor.Monitor@30c665da [INFO] [pekkoDeadLetter][10/29/2024 01:13:38.595] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower552] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:38,596 AM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower552] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:38,598 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Recovery completed in 3.757 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:38,598 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower552 with journal batch size 1000 01:13:38,599 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: SnapshotOffer called. 01:13:38,599 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Moving last applied index from -1 to -1 01:13:38,599 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=1, votedFor=null 01:13:38,599 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 01:13:38,599 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower552 in 59.53 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:38,600 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Recovery completed in 858.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:38,698 AM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower553] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [10/29/2024 01:13:38.698] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower553]: org.opendaylight.controller.cluster.common.actor.Monitor@494222ee [INFO] [pekkoDeadLetter][10/29/2024 01:13:38.698] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower553] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:38,699 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower553 with journal batch size 1000 01:13:38,700 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: SnapshotOffer called. 01:13:38,700 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Moving last applied index from -1 to -1 01:13:38,700 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Set currentTerm=1, votedFor=null 01:13:38,700 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Updated server config: isVoting: false, peers: [PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 01:13:38,700 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower553 in 88.55 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:38,701 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Recovery completed in 862.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:38.801] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader550]: org.opendaylight.controller.cluster.common.actor.Monitor@650b1e4b [INFO] [pekkoDeadLetter][10/29/2024 01:13:38.801] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader550] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:38,801 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader550 with journal batch size 1000 01:13:38,801 AM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader550] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:38,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: SnapshotOffer called. 01:13:38,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Moving last applied index from -1 to -1 01:13:38,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Set currentTerm=1, votedFor=null 01:13:38,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 01:13:38,802 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader550 in 74.92 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:38,803 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Recovery completed in 1.096 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:39,171 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:13:39,171 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.219:17355, Removed) 01:13:39,171 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Received ElectionTimeout - switching to Candidate 01:13:39,171 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Election: Candidate has following voting peers: [follower552, follower551] 01:13:39,171 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Set currentTerm=2, votedFor=leader550 01:13:39,171 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Starting new election term 2 01:13:39,171 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] to peer follower552 01:13:39,171 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] to peer follower551 01:13:39,171 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:39,171 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:13:39,171 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Cluster state: [] 01:13:39,171 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:39,171 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:39,171 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=2, votedFor=null 01:13:39,172 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:39,172 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): In requestVote: RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:39,172 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=2, votedFor=leader550 01:13:39,172 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Cluster state: [] 01:13:39,172 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:39,172 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:39,172 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=2, votedFor=null 01:13:39,172 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:39,172 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): In requestVote: RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:39,172 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=2, votedFor=leader550 01:13:39,172 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:39,173 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:13:39,173 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:13:39,173 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Election: Leader has following peers: [follower553, follower552, follower551] 01:13:39,173 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,173 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,174 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,174 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553 (Follower): Term 2 in "AppendEntries [leaderId=leader550, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:13:39,174 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Set currentTerm=2, votedFor=null 01:13:39,174 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:13:39,174 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower553: Last sync leader does not match current leader leader550, need to catch up to -1 01:13:39,174 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower552: Last sync leader does not match current leader leader550, need to catch up to -1 01:13:39,174 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower553 updated: matchIndex: -1, nextIndex: 0 01:13:39,174 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower551: Last sync leader does not match current leader leader550, need to catch up to -1 01:13:39,174 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,174 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower552 updated: matchIndex: -1, nextIndex: 0 01:13:39,174 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,174 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower551 updated: matchIndex: -1, nextIndex: 0 01:13:39,174 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:39.175] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower553] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7844751 01:13:39,175 AM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][10/29/2024 01:13:39.175] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [10/29/2024 01:13:39.176] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower551] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4286d9c5 [WARN] [10/29/2024 01:13:39.176] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower552] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2e77f890 [INFO] [pekkoDeadLetter][10/29/2024 01:13:39.176] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:39,176 AM [test-pekko.actor.default-dispatcher-11] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:39,178 AM [test-pekko.actor.default-dispatcher-16] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][10/29/2024 01:13:39.178] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:39,208 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:13:39,208 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader550 starting 01:13:39,211 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: onShutDown 01:13:39,211 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Initiating leader transfer 01:13:39,212 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader550: pauseLeader successfully completed - doing transfer 01:13:39,212 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Attempting to transfer leadership 01:13:39,212 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 01:13:39,212 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,212 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 01:13:39,212 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,212 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,212 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,212 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,212 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,212 AM [test-pekko.actor.default-dispatcher-15] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:39,212 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 01:13:39,212 AM [test-pekko.actor.default-dispatcher-15] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:39,213 AM [test-pekko.actor.default-dispatcher-17] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [10/29/2024 01:13:39.213] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower551] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@730efa83 [WARN] [10/29/2024 01:13:39.214] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower552] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5615857c [INFO] [pekkoDeadLetter][10/29/2024 01:13:39.212] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][10/29/2024 01:13:39.212] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [10/29/2024 01:13:39.215] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower553] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@22a9268a [INFO] [pekkoDeadLetter][10/29/2024 01:13:39.213] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:13:39,291 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,291 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,291 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,411 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,411 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,411 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,531 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader550: leader transfer aborted 01:13:39,531 AM [test-pekko.actor.default-dispatcher-11] [WARN] RaftActorLeadershipTransferCohort - leader550: Failed to transfer leadership in 319.6 ms 01:13:39,532 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: leader transfer failed - sending PoisonPill 01:13:39,532 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Leadership transfer expired 01:13:39,532 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,532 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,532 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:13:39,533 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader550 ending [WARN] [10/29/2024 01:13:39.533] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/leader550] received dead letter from TestActor[pekko://test/user/follower552]: AppendEntriesReply [term=2, success=true, followerId=follower552, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [10/29/2024 01:13:39.534] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/leader550] received dead letter from TestActor[pekko://test/user/follower553]: AppendEntriesReply [term=2, success=true, followerId=follower553, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:13:39,535 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 01:13:39,535 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower551-notifier554#-347155788] 01:13:39,546 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower551-collector555#-1089777913] 01:13:39,546 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower551] 01:13:39,546 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower552-notifier556#570891111] 01:13:39,547 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower552-collector557#815417452] 01:13:39,547 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower552] 01:13:39,547 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower553-notifier558#-356659293] 01:13:39,547 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower553-collector559#-1957477772] 01:13:39,547 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower553] 01:13:39,547 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader550-notifier560#-363891742] 01:13:39,547 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader550-collector561#-1179843616] 01:13:39,547 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader550] 01:13:39,548 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 01:13:39,548 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/29/2024 01:13:39.551] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower563]: org.opendaylight.controller.cluster.common.actor.Monitor@3755c727 01:13:39,552 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower563 with journal batch size 1000 01:13:39,553 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: SnapshotOffer called. 01:13:39,553 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Moving last applied index from -1 to -1 01:13:39,553 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=1, votedFor=null 01:13:39,553 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 01:13:39,553 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower564 with journal batch size 1000 01:13:39,553 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower563 in 68.91 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [10/29/2024 01:13:39.554] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower564]: org.opendaylight.controller.cluster.common.actor.Monitor@55311825 01:13:39,555 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower565 with journal batch size 1000 [WARN] [10/29/2024 01:13:39.556] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower565]: org.opendaylight.controller.cluster.common.actor.Monitor@15530340 [WARN] [10/29/2024 01:13:39.557] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader562]: org.opendaylight.controller.cluster.common.actor.Monitor@61996faf 01:13:39,557 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader562 with journal batch size 1000 01:13:39,558 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: SnapshotOffer called. 01:13:39,558 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: SnapshotOffer called. 01:13:39,558 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Moving last applied index from -1 to -1 01:13:39,558 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Moving last applied index from -1 to -1 01:13:39,558 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=1, votedFor=null 01:13:39,558 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=1, votedFor=null 01:13:39,558 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Updated server config: isVoting: true, peers: [PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 01:13:39,558 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Updated server config: isVoting: false, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING]] 01:13:39,558 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader562 in 35.73 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:39,558 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower565 in 70.91 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:39,559 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: SnapshotOffer called. 01:13:39,559 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Moving last applied index from -1 to -1 01:13:39,559 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=1, votedFor=null 01:13:39,559 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 01:13:39,559 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Recovery completed in 5.576 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:39,559 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Recovery completed in 772.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:39,559 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Recovery completed in 792.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:39,559 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower564 in 26.73 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:39,561 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Recovery completed in 2.439 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:39,881 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:13:39,881 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.219:17355, Removed) 01:13:39,881 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Received ElectionTimeout - switching to Candidate 01:13:39,881 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Election: Candidate has following voting peers: [follower564, follower563] 01:13:39,882 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=2, votedFor=leader562 01:13:39,882 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Starting new election term 2 01:13:39,882 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] to peer follower564 01:13:39,882 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] to peer follower563 01:13:39,882 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:39,882 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:13:39,882 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Cluster state: [] 01:13:39,882 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:39,882 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:39,882 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:39,882 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=2, votedFor=null 01:13:39,882 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 01:13:39,883 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:39,883 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:39,883 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=2, votedFor=null 01:13:39,883 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): In requestVote: RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:39,883 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=2, votedFor=leader562 01:13:39,883 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:39,883 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:39,883 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=2, votedFor=leader562 01:13:39,883 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:39,884 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:13:39,884 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:13:39,884 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Election: Leader has following peers: [follower564, follower563, follower565] 01:13:39,885 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,885 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,885 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,885 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:13:39,885 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 2 in "AppendEntries [leaderId=leader562, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:13:39,885 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader leader562, need to catch up to -1 01:13:39,885 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=2, votedFor=null 01:13:39,885 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower564: Last sync leader does not match current leader leader562, need to catch up to -1 01:13:39,885 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 01:13:39,885 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader leader562, need to catch up to -1 01:13:39,885 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,885 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower564 updated: matchIndex: -1, nextIndex: 0 01:13:39,885 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,885 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 01:13:39,885 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:39.885] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@58756908 [WARN] [10/29/2024 01:13:39.886] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@189101c0 [WARN] [10/29/2024 01:13:39.886] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7448be0e 01:13:39,912 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:13:39,912 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:13:39,913 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:13:39,913 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:13:39,913 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:13:39,914 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: onRequestLeadership RequestLeadership [requestedFollowerId=follower564, replyTo=Actor[pekko://test/user/$a#1815058650]] 01:13:39,915 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Initiating leader transfer 01:13:39,915 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader562: pauseLeader successfully completed - doing transfer 01:13:39,915 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 01:13:39,915 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Attempting to transfer leadership 01:13:39,915 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,915 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 01:13:39,915 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,915 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 01:13:39,916 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,916 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:13:39,916 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:13:39,916 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) [WARN] [10/29/2024 01:13:39.916] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1694fc6 [WARN] [10/29/2024 01:13:39.916] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3c005bd6 [WARN] [10/29/2024 01:13:39.916] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@450da105 01:13:39,916 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): tryToCompleteLeadershipTransfer: followerId: follower564, matchIndex: -1, lastIndex: -1, isVoting: true 01:13:39,916 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Follower's log matches - sending ElectionTimeout 01:13:39,916 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,917 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,917 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,917 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:13:39,917 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leader transfer complete 01:13:39,917 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader transfer complete - waiting for new leader 01:13:39,917 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:13:39,917 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:13:39,917 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Received TimeoutNow - switching to Candidate 01:13:39,917 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:13:39,917 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Election: Candidate has following voting peers: [leader562, follower563] 01:13:39,917 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=3, votedFor=follower564 01:13:39,917 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Starting new election term 3 01:13:39,917 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] to peer leader562 01:13:39,917 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] to peer follower563 01:13:39,917 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Cluster state: [] 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=3, votedFor=null 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leadership transfer in progress - processing RequestVote 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): In requestVote: RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=3, votedFor=follower564 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader) :- Switching from behavior Leader to Follower, election term: 3 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 2 - updating term 01:13:39,918 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=3, votedFor=null 01:13:39,919 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:39,919 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=3, votedFor=follower564 01:13:39,919 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:13:39,919 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 01:13:39,919 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 01:13:39,920 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Election: Leader has following peers: [leader562, follower563, follower565] 01:13:39,920 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,920 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,920 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,920 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - leader562: Last sync leader does not match current leader follower564, need to catch up to -1 01:13:39,920 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader changed to follower564 01:13:39,920 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 01:13:39,920 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader follower564, need to catch up to -1 01:13:39,920 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 3 in "AppendEntries [leaderId=follower564, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 01:13:39,920 AM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorLeadershipTransferCohort - leader562: Successfully transferred leadership to follower564 in 4.740 ms 01:13:39,920 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=3, votedFor=null 01:13:39,920 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Leadership transferred successfully to follower564 [WARN] [10/29/2024 01:13:39.920] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2326e014 01:13:39,920 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader follower564, need to catch up to -1 01:13:39,920 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader562 updated: matchIndex: -1, nextIndex: 0 01:13:39,920 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:39.920] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/leader562] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@122f92b0 01:13:39,920 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 01:13:39,920 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:39,920 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 01:13:39,921 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:39.921] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@86157f8 01:13:39,967 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 01:13:39,967 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower563-notifier566#993445384] 01:13:39,978 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower563-collector567#1175099965] 01:13:39,979 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower563] 01:13:39,979 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower564-notifier568#798012305] 01:13:39,979 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower564-collector569#-1308760771] 01:13:39,979 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower564] 01:13:39,979 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower565-notifier570#-322296692] 01:13:39,980 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower565-collector571#964192983] 01:13:39,980 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower565] 01:13:39,980 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader562-notifier572#-760710981] 01:13:39,980 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader562-collector573#-261424429] 01:13:39,980 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader562] 01:13:39,981 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$a#1815058650] 01:13:39,982 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 01:13:39,982 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/29/2024 01:13:39.984] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower575]: org.opendaylight.controller.cluster.common.actor.Monitor@4e32bbee 01:13:39,985 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower575 with journal batch size 1000 01:13:39,986 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: SnapshotOffer called. 01:13:39,986 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Moving last applied index from -1 to -1 01:13:39,986 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=1, votedFor=null [WARN] [10/29/2024 01:13:39.986] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower576]: org.opendaylight.controller.cluster.common.actor.Monitor@3eaa5ac0 01:13:39,987 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower576 with journal batch size 1000 01:13:39,987 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 01:13:39,987 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: SnapshotOffer called. 01:13:39,987 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower575 in 1.159 ms: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:39,987 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Moving last applied index from -1 to -1 01:13:39,987 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=1, votedFor=null 01:13:39,987 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING]] 01:13:39,987 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower576 in 104.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:39,988 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Recovery completed in 805.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:39,989 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Recovery completed in 2.898 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:39.989] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower577]: org.opendaylight.controller.cluster.common.actor.Monitor@39b94d5e 01:13:39,990 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower577 with journal batch size 1000 01:13:39,990 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: SnapshotOffer called. 01:13:39,990 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Moving last applied index from -1 to -1 01:13:39,990 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Set currentTerm=1, votedFor=null 01:13:39,990 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Updated server config: isVoting: false, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 01:13:39,990 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower577 in 78.24 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:39,991 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Recovery completed in 788.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:39.992] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader574]: org.opendaylight.controller.cluster.common.actor.Monitor@6187e6a5 01:13:39,992 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader574 with journal batch size 1000 01:13:39,992 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: SnapshotOffer called. 01:13:39,992 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Moving last applied index from -1 to -1 01:13:39,992 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Set currentTerm=1, votedFor=null 01:13:39,992 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Updated server config: isVoting: true, peers: [PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 01:13:39,992 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader574 in 40.70 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:39,993 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Recovery completed in 531.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:40,351 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:13:40,351 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.219:17355, Removed) 01:13:40,351 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Received ElectionTimeout - switching to Candidate 01:13:40,351 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Election: Candidate has following voting peers: [follower575, follower576] 01:13:40,351 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Set currentTerm=2, votedFor=leader574 01:13:40,351 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Starting new election term 2 01:13:40,351 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] to peer follower575 01:13:40,351 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] to peer follower576 01:13:40,351 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:13:40,351 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:40,352 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Cluster state: [] 01:13:40,352 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:40,352 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:40,352 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Cluster state: [] 01:13:40,352 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:40,352 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:40,352 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=2, votedFor=null 01:13:40,352 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): In requestVote: RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:40,352 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=2, votedFor=leader574 01:13:40,352 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:40,353 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:13:40,353 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:13:40,353 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Election: Leader has following peers: [follower575, follower577, follower576] 01:13:40,353 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:40,353 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:40,353 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:40,353 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:13:40,353 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577 (Follower): Term 2 in "AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:13:40,353 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Set currentTerm=2, votedFor=null 01:13:40,353 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower576: Last sync leader does not match current leader leader574, need to catch up to -1 01:13:40,353 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower576 updated: matchIndex: -1, nextIndex: 0 01:13:40,353 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:40,353 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower577: Last sync leader does not match current leader leader574, need to catch up to -1 01:13:40,353 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower577 updated: matchIndex: -1, nextIndex: 0 [WARN] [10/29/2024 01:13:40.353] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower576] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6ef059ac 01:13:40,353 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:40,352 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:40,354 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=2, votedFor=null 01:13:40,354 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): In requestVote: RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:40,354 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=2, votedFor=leader574 01:13:40,354 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] [WARN] [10/29/2024 01:13:40.354] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower577] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3c2119fe 01:13:40,355 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower575: Last sync leader does not match current leader leader574, need to catch up to -1 01:13:40,355 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower575 updated: matchIndex: -1, nextIndex: 0 [WARN] [10/29/2024 01:13:40.355] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower575] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@441c5c5 01:13:40,355 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:40,396 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:13:40,396 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:13:40,397 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:13:40,397 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower576 starting 01:13:40,397 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: onShutDown 01:13:40,398 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower576 ending 01:13:40,398 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:13:40,398 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:13:40,398 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: onRequestLeadership RequestLeadership [requestedFollowerId=follower576, replyTo=Actor[pekko://test/user/$b#-1881049093]] 01:13:40,398 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Initiating leader transfer 01:13:40,398 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] 01:13:40,398 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader574: pauseLeader successfully completed - doing transfer 01:13:40,398 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Attempting to transfer leadership 01:13:40,398 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] 01:13:40,398 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:40.398] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: LeaderTransitioning [leaderId=leader574] 01:13:40,398 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:40,398 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:40,399 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [10/29/2024 01:13:40.399] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [10/29/2024 01:13:40.399] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower575] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1dbad54a 01:13:40,399 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [10/29/2024 01:13:40.399] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower577] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7da104e6 [WARN] [10/29/2024 01:13:40.472] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:13:40,591 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 01:13:40,591 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [10/29/2024 01:13:40.591] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:13:40,711 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader574: leader transfer aborted 01:13:40,711 AM [test-pekko.actor.default-dispatcher-15] [WARN] RaftActorLeadershipTransferCohort - leader574: Failed to transfer leadership in 313.3 ms 01:13:40,711 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: LeadershipTransfer request from follower576 failed 01:13:40,712 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Leadership transfer expired 01:13:40,713 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 01:13:40,713 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [10/29/2024 01:13:40.713] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:13:40,752 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 01:13:40,753 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower575-notifier578#1442293735] 01:13:40,764 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower575-collector579#540626068] 01:13:40,764 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower575] 01:13:40,765 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower576-notifier580#1286667626] 01:13:40,765 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower576-collector581#-150367626] 01:13:40,765 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower576] 01:13:40,765 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower577-notifier582#-1283546105] 01:13:40,765 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower577-collector583#-728740046] 01:13:40,766 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower577] 01:13:40,766 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader574-notifier584#1842409191] 01:13:40,766 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader574-collector585#-759196803] 01:13:40,766 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader574] 01:13:40,766 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$b#-1881049093] 01:13:40,768 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 01:13:40,768 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/29/2024 01:13:40.772] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower587]: org.opendaylight.controller.cluster.common.actor.Monitor@53de69c5 01:13:40,773 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower587 with journal batch size 1000 01:13:40,773 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: SnapshotOffer called. 01:13:40,773 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to -1 01:13:40,773 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=1, votedFor=null 01:13:40,774 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING]] 01:13:40,774 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower587 in 85.82 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [10/29/2024 01:13:40.774] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower588]: org.opendaylight.controller.cluster.common.actor.Monitor@877aef 01:13:40,775 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Recovery completed in 1.142 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:40,775 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower588 with journal batch size 1000 01:13:40,776 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: SnapshotOffer called. 01:13:40,776 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to -1 01:13:40,776 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=1, votedFor=null 01:13:40,776 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 01:13:40,776 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower588 in 102.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:40,777 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Recovery completed in 986.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:40.777] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower589]: org.opendaylight.controller.cluster.common.actor.Monitor@6f56dbf2 01:13:40,778 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower589 with journal batch size 1000 01:13:40,778 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: SnapshotOffer called. 01:13:40,778 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to -1 01:13:40,778 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=1, votedFor=null 01:13:40,778 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Updated server config: isVoting: false, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 01:13:40,778 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower589 in 46.83 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:40,779 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Recovery completed in 454.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:40.779] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader586]: org.opendaylight.controller.cluster.common.actor.Monitor@55b2edfe 01:13:40,780 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader586 with journal batch size 1000 01:13:40,780 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: SnapshotOffer called. 01:13:40,780 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to -1 01:13:40,780 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=1, votedFor=null 01:13:40,780 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Updated server config: isVoting: true, peers: [PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 01:13:40,780 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader586 in 92.62 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:40,781 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Recovery completed in 609.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:41,201 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:13:41,201 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.219:17355, Removed) 01:13:41,201 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Received ElectionTimeout - switching to Candidate 01:13:41,201 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Election: Candidate has following voting peers: [follower588, follower587] 01:13:41,201 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=2, votedFor=leader586 01:13:41,201 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Starting new election term 2 01:13:41,201 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] to peer follower588 01:13:41,202 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] to peer follower587 01:13:41,202 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:13:41,202 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:41,202 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 01:13:41,202 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:41,202 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:41,202 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=2, votedFor=null 01:13:41,202 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:41,202 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:41,202 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=2, votedFor=leader586 01:13:41,202 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Cluster state: [] 01:13:41,202 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:41,202 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:41,202 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:41,202 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=2, votedFor=null 01:13:41,203 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): In requestVote: RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:41,203 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=2, votedFor=leader586 01:13:41,203 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:41,203 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:13:41,203 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:13:41,203 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Election: Leader has following peers: [follower589, follower588, follower587] 01:13:41,203 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:41,203 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:41,203 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:41,203 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:13:41,203 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 2 in "AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:13:41,204 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=2, votedFor=null 01:13:41,204 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower587: Last sync leader does not match current leader leader586, need to catch up to -1 01:13:41,204 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader leader586, need to catch up to -1 01:13:41,204 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: -1, nextIndex: 0 [WARN] [10/29/2024 01:13:41.204] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@31f3f9c2 01:13:41,204 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:41,204 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: -1, nextIndex: 0 01:13:41,204 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:41,204 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader leader586, need to catch up to -1 01:13:41,204 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: -1, nextIndex: 0 [WARN] [10/29/2024 01:13:41.205] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@529e8eb 01:13:41,205 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:41.205] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@d78de1a 01:13:41,235 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:13:41,235 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 01:13:41,235 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:41,235 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower589 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower589: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:41,236 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:41,236 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Number of entries to be appended = 1 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,236 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower587 01:13:41,236 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Append entry to log zero 01:13:41,236 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower587: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:41,236 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Log size is now 1 01:13:41,236 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower589, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:13:41,236 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 01:13:41,236 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 01:13:41,236 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Number of entries to be appended = 1 01:13:41,236 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:13:41,236 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Append entry to log zero 01:13:41,236 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,236 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:41,236 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Log size is now 1 01:13:41,236 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower587, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:13:41,237 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 01:13:41,237 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: 0, nextIndex: 1 01:13:41,237 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,237 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:41,237 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:13:41,238 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Setting last applied to 0 01:13:41,238 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to 0 01:13:41,238 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Applying state for log index 0 data zero 01:13:41,238 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:41,238 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: applyState called: zero 01:13:41,238 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persisting ApplyJournalEntries with index=0 01:13:41,238 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [10/29/2024 01:13:41.238] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@25555339 [WARN] [10/29/2024 01:13:41.238] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@799a6644 01:13:41,320 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:13:41,320 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:41,441 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:13:41,441 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:41,442 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Commit index set to 0 01:13:41,442 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:13:41,442 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:13:41,442 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Setting last applied to 0 01:13:41,442 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to 0 01:13:41,442 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Applying state for log index 0 data zero 01:13:41,442 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Commit index set to 0 01:13:41,442 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:13:41,442 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:13:41,442 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Setting last applied to 0 01:13:41,442 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to 0 01:13:41,442 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: applyState called: zero 01:13:41,442 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Applying state for log index 0 data zero 01:13:41,442 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: applyState called: zero 01:13:41,442 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Persisting ApplyJournalEntries with index=0 01:13:41,442 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Persisting ApplyJournalEntries with index=0 01:13:41,488 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 01:13:41,488 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 01:13:41,489 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: onShutDown 01:13:41,489 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Initiating leader transfer 01:13:41,489 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader586: pauseLeader successfully completed - doing transfer 01:13:41,489 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 01:13:41,489 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Attempting to transfer leadership 01:13:41,489 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 01:13:41,489 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,489 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,489 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:13:41,489 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,489 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:13:41,490 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower589, matchIndex: 0, lastIndex: 0, isVoting: false 01:13:41,490 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:13:41,490 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower587, matchIndex: 0, lastIndex: 0, isVoting: true 01:13:41,490 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Follower's log matches - sending ElectionTimeout 01:13:41,490 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,490 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,490 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:13:41,490 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,490 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leader transfer complete 01:13:41,490 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader transfer complete - waiting for new leader 01:13:41,490 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:13:41,490 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:13:41,490 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:13:41,490 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Received TimeoutNow - switching to Candidate 01:13:41,491 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Election: Candidate has following voting peers: [leader586, follower588] 01:13:41,491 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=3, votedFor=follower587 01:13:41,491 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Starting new election term 3 01:13:41,491 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] to peer leader586 01:13:41,491 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] to peer follower588 01:13:41,491 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:41,491 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:13:41,491 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Cluster state: [] 01:13:41,491 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:41,491 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 01:13:41,491 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2]" message is greater than leader's term 2 - switching to Follower 01:13:41,491 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=3, votedFor=null 01:13:41,491 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 01:13:41,491 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 01:13:41,491 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2]" message is greater than follower's term 2 - updating term 01:13:41,491 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leadership transfer in progress - processing RequestVote 01:13:41,491 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=3, votedFor=null 01:13:41,491 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): In requestVote: RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 01:13:41,491 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=3, votedFor=follower587 01:13:41,491 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:13:41,491 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:41,491 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=3, votedFor=follower587 01:13:41,491 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:13:41,492 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader) :- Switching from behavior Leader to Follower, election term: 3 01:13:41,493 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 01:13:41,493 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 01:13:41,493 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Election: Leader has following peers: [leader586, follower589, follower588] 01:13:41,493 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,493 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,493 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,493 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - leader586: Last sync leader does not match current leader follower587, need to catch up to -1 01:13:41,493 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 01:13:41,493 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader changed to follower587 01:13:41,493 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 3 in "AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 01:13:41,493 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=3, votedFor=null 01:13:41,493 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorLeadershipTransferCohort - leader586: Successfully transferred leadership to follower587 in 3.983 ms 01:13:41,493 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: leader transfer succeeded - sending PoisonPill 01:13:41,493 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader follower587, need to catch up to -1 [WARN] [10/29/2024 01:13:41.493] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/leader586] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@38675c9d 01:13:41,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader586 updated: matchIndex: 0, nextIndex: 1 01:13:41,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:41.494] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@47d6012b 01:13:41,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 [WARN] [10/29/2024 01:13:41.494] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/leader586] received dead letter from TestActor[pekko://test/user/follower587]: AppendEntries [leaderId=follower587, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:13:41,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:41,494 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) [WARN] [10/29/2024 01:13:41.494] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@67f36559 01:13:42,11 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader follower587, need to catch up to -1 01:13:42,11 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) [WARN] [10/29/2024 01:13:42.011] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader586] received dead letter from TestActor[pekko://test/user/follower587]: AppendEntries [leaderId=follower587, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [10/29/2024 01:13:42.011] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4ea2c040 01:13:42,12 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): sendAppendEntries: 0 is present for follower follower588 01:13:42,12 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries: AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Number of entries to be appended = 1 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Append entry to log zero 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Log size is now 1 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Commit index set to 0 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries returning : AppendEntriesReply [term=3, success=true, followerId=follower588, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Setting last applied to 0 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to 0 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Applying state for log index 0 data zero 01:13:42,12 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: 0, nextIndex: 1 01:13:42,12 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:42,12 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: applyState called: zero 01:13:42,14 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:42,14 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Persisting ApplyJournalEntries with index=0 [WARN] [10/29/2024 01:13:42.014] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@129e362f 01:13:42,43 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 01:13:42,43 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 01:13:42,43 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower588 starting 01:13:42,44 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onShutDown 01:13:42,44 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower588 ending 01:13:42,44 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 01:13:42,44 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587-notifier590#718950375] 01:13:42,55 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587-collector591#-1538114891] 01:13:42,55 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower587] 01:13:42,57 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower588-notifier592#1592270637] 01:13:42,57 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower588-collector593#498028726] 01:13:42,57 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower588] 01:13:42,57 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower589-notifier594#1100077676] 01:13:42,57 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower589-collector595#572090786] 01:13:42,58 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower589] 01:13:42,58 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader586-notifier596#-498806195] 01:13:42,58 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader586-collector597#1415295662] 01:13:42,58 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader586] 01:13:42,59 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [10/29/2024 01:13:42.061] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader598]: org.opendaylight.controller.cluster.common.actor.Monitor@38fc00bf 01:13:42,61 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader598 with journal batch size 1000 01:13:42,63 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:42,63 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:13:42,63 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower): Received TimeoutNow - switching to Candidate 01:13:42,63 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Election: Candidate has following voting peers: [] 01:13:42,63 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Set currentTerm=1, votedFor=leader598 01:13:42,63 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Starting new election term 1 01:13:42,63 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 01:13:42,64 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Received ElectionTimeout 01:13:42,64 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Leader): Election: Leader has following peers: [] 01:13:42,64 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 01:13:42,64 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader598 starting 01:13:42,64 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: onShutDown 01:13:42,65 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader598 ending 01:13:42,65 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 01:13:42,65 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader598-collector602#-1805792904] 01:13:42,75 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader598] 01:13:42,76 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 01:13:42,76 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/29/2024 01:13:42.078] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower604]: org.opendaylight.controller.cluster.common.actor.Monitor@5ae949c1 01:13:42,80 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower604 with journal batch size 1000 01:13:42,81 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: SnapshotOffer called. 01:13:42,81 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to -1 01:13:42,81 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=1, votedFor=null 01:13:42,81 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 01:13:42,81 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower604 in 120.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:42,82 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Recovery completed in 1.093 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:42.181] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower605]: org.opendaylight.controller.cluster.common.actor.Monitor@322ce159 01:13:42,182 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower605 with journal batch size 1000 01:13:42,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: SnapshotOffer called. 01:13:42,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Moving last applied index from -1 to -1 01:13:42,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=1, votedFor=null 01:13:42,184 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Updated server config: isVoting: true, peers: [PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 01:13:42,184 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower605 in 225.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:42,185 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Recovery completed in 1.153 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:42.185] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower606]: org.opendaylight.controller.cluster.common.actor.Monitor@38a460a4 01:13:42,186 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower606 with journal batch size 1000 01:13:42,186 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: SnapshotOffer called. 01:13:42,186 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to -1 01:13:42,186 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Set currentTerm=1, votedFor=null 01:13:42,186 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Updated server config: isVoting: false, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 01:13:42,186 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower606 in 113.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:42,187 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Recovery completed in 794.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:42.188] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader603]: org.opendaylight.controller.cluster.common.actor.Monitor@42eb7df4 01:13:42,188 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader603 with journal batch size 1000 01:13:42,189 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: SnapshotOffer called. 01:13:42,189 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to -1 01:13:42,189 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Set currentTerm=1, votedFor=null 01:13:42,189 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING]] 01:13:42,189 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader603 in 48.12 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:42,189 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Recovery completed in 493.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:42,560 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:13:42,561 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.219:17355, Removed) 01:13:42,561 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Received ElectionTimeout - switching to Candidate 01:13:42,561 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Election: Candidate has following voting peers: [follower605, follower604] 01:13:42,561 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Set currentTerm=2, votedFor=leader603 01:13:42,561 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Starting new election term 2 01:13:42,561 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] to peer follower605 01:13:42,561 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] to peer follower604 01:13:42,561 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:13:42,561 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:42,561 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:42,561 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Cluster state: [] 01:13:42,561 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:42,561 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:42,561 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Cluster state: [] 01:13:42,561 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=2, votedFor=null 01:13:42,561 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:42,561 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:42,561 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=2, votedFor=null 01:13:42,561 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): In requestVote: RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:42,561 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=2, votedFor=leader603 01:13:42,561 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): In requestVote: RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:42,561 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=2, votedFor=leader603 01:13:42,561 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:42,562 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:42,562 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:13:42,563 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:13:42,563 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Election: Leader has following peers: [follower605, follower604, follower606] 01:13:42,563 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:42,563 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:42,563 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:42,563 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:13:42,563 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Term 2 in "AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:13:42,563 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Set currentTerm=2, votedFor=null 01:13:42,563 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower605: Last sync leader does not match current leader leader603, need to catch up to -1 01:13:42,563 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower604: Last sync leader does not match current leader leader603, need to catch up to -1 01:13:42,563 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower606: Last sync leader does not match current leader leader603, need to catch up to -1 01:13:42,563 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower605 updated: matchIndex: -1, nextIndex: 0 01:13:42,563 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:42,564 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: -1, nextIndex: 0 [WARN] [10/29/2024 01:13:42.563] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower605] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@491c9e30 01:13:42,564 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:42,564 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: -1, nextIndex: 0 01:13:42,564 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:42.564] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower604] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@ec947da [WARN] [10/29/2024 01:13:42.564] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower606] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5bba5973 01:13:42,594 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:13:42,594 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:13:42,594 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:13:42,594 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower604 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower604: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower606 01:13:42,595 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower606: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:42,595 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:42,595 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Number of entries to be appended = 1 01:13:42,595 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:13:42,595 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:42,595 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Append entry to log zero 01:13:42,595 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:42,595 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Number of entries to be appended = 1 01:13:42,595 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:13:42,595 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Append entry to log zero 01:13:42,595 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:42,595 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Log size is now 1 01:13:42,595 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Log size is now 1 01:13:42,595 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower604, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:13:42,596 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower606, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:13:42,596 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 01:13:42,596 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: 0, nextIndex: 1 01:13:42,596 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 01:13:42,596 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:42,596 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: 0, nextIndex: 1 01:13:42,596 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:42,597 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:42,597 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:13:42,597 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Setting last applied to 0 01:13:42,597 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to 0 01:13:42,597 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Applying state for log index 0 data zero 01:13:42,597 AM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: applyState called: zero 01:13:42,597 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:13:42,597 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persisting ApplyJournalEntries with index=0 01:13:42,597 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [10/29/2024 01:13:42.597] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower604] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@38337c8e [WARN] [10/29/2024 01:13:42.597] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower606] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@58b228b9 01:13:42,681 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:13:42,681 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:42,801 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:13:42,801 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:13:42,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Commit index set to 0 01:13:42,802 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Commit index set to 0 01:13:42,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:13:42,802 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:13:42,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:13:42,802 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:13:42,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Setting last applied to 0 01:13:42,802 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Setting last applied to 0 01:13:42,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to 0 01:13:42,802 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to 0 01:13:42,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Applying state for log index 0 data zero 01:13:42,802 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Applying state for log index 0 data zero 01:13:42,802 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: applyState called: zero 01:13:42,802 AM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: applyState called: zero 01:13:42,802 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Persisting ApplyJournalEntries with index=0 01:13:42,802 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Persisting ApplyJournalEntries with index=0 01:13:42,846 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 01:13:42,846 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:13:42,846 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:13:42,846 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: onRequestLeadership RequestLeadership [requestedFollowerId=follower605, replyTo=Actor[pekko://test/user/$c#-804793794]] 01:13:42,846 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Initiating leader transfer 01:13:42,847 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 01:13:42,847 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader603: pauseLeader successfully completed - doing transfer 01:13:42,847 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Attempting to transfer leadership 01:13:42,847 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 01:13:42,847 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:42,847 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:13:42,847 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:42,847 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 01:13:42,847 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:13:42,847 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:13:42,847 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:13:43,41 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:13:43,41 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:13:43,161 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader603: leader transfer aborted 01:13:43,161 AM [test-pekko.actor.default-dispatcher-17] [WARN] RaftActorLeadershipTransferCohort - leader603: Failed to transfer leadership in 314.4 ms 01:13:43,161 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: LeadershipTransfer request from follower605 failed 01:13:43,161 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Leadership transfer expired 01:13:43,161 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:13:43,161 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:13:43,200 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 01:13:43,200 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower604-notifier607#1375301841] 01:13:43,211 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower604-collector608#-1177260137] 01:13:43,211 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower604] 01:13:43,211 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower605-notifier609#-280218800] 01:13:43,211 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower605-collector610#-1113513776] 01:13:43,212 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower605] 01:13:43,212 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower606-notifier611#253604643] 01:13:43,212 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower606-collector612#1739011460] 01:13:43,212 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower606] 01:13:43,212 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader603-notifier613#1678907628] 01:13:43,212 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader603-collector614#1771600334] 01:13:43,213 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader603] 01:13:43,213 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$c#-804793794] 01:13:43,214 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 01:13:43,214 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/29/2024 01:13:43.216] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower616]: org.opendaylight.controller.cluster.common.actor.Monitor@406dc839 01:13:43,216 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower616 with journal batch size 1000 01:13:43,217 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: SnapshotOffer called. 01:13:43,217 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Moving last applied index from -1 to -1 01:13:43,217 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=1, votedFor=null 01:13:43,217 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Updated server config: isVoting: true, peers: [PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 01:13:43,217 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower616 in 108.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:43,218 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Recovery completed in 966.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:43.318] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower617]: org.opendaylight.controller.cluster.common.actor.Monitor@6f8c4bc3 01:13:43,319 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower617 with journal batch size 1000 01:13:43,320 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: SnapshotOffer called. 01:13:43,320 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Moving last applied index from -1 to -1 01:13:43,320 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=1, votedFor=null 01:13:43,320 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 01:13:43,320 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower617 in 104.4 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:43,321 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Recovery completed in 1.031 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:43.420] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower618]: org.opendaylight.controller.cluster.common.actor.Monitor@3d267395 01:13:43,420 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower618 with journal batch size 1000 01:13:43,420 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: SnapshotOffer called. 01:13:43,421 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Moving last applied index from -1 to -1 01:13:43,421 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Set currentTerm=1, votedFor=null 01:13:43,421 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Updated server config: isVoting: false, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 01:13:43,421 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower618 in 61.05 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:43,421 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Recovery completed in 629.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/29/2024 01:13:43.422] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader615]: org.opendaylight.controller.cluster.common.actor.Monitor@49c04430 01:13:43,422 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader615 with journal batch size 1000 01:13:43,423 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: SnapshotOffer called. 01:13:43,423 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Moving last applied index from -1 to -1 01:13:43,423 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=1, votedFor=null 01:13:43,423 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING]] 01:13:43,423 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader615 in 44.32 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:13:43,423 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Recovery completed in 444.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:13:43,800 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:13:43,801 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.219:17355, Removed) 01:13:43,801 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Received ElectionTimeout - switching to Candidate 01:13:43,801 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Election: Candidate has following voting peers: [follower616, follower617] 01:13:43,801 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=2, votedFor=leader615 01:13:43,801 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Starting new election term 2 01:13:43,801 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] to peer follower616 01:13:43,801 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] to peer follower617 01:13:43,801 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:43,801 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:43,801 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Cluster state: [] 01:13:43,801 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:43,801 AM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:43,801 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=2, votedFor=null 01:13:43,801 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Cluster state: [] 01:13:43,801 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): In requestVote: RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:43,801 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=2, votedFor=leader615 01:13:43,801 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:43,801 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:13:43,801 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:43,801 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=2, votedFor=null 01:13:43,801 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): In requestVote: RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:43,802 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=2, votedFor=leader615 01:13:43,802 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:13:43,801 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:13:43,802 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:13:43,802 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Election: Leader has following peers: [follower616, follower618, follower617] 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:13:43,803 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower617: Last sync leader does not match current leader leader615, need to catch up to -1 01:13:43,803 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618 (Follower): Term 2 in "AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:13:43,803 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower616: Last sync leader does not match current leader leader615, need to catch up to -1 01:13:43,803 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Set currentTerm=2, votedFor=null [WARN] [10/29/2024 01:13:43.803] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower617] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3e7b8d25 01:13:43,803 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower618: Last sync leader does not match current leader leader615, need to catch up to -1 [WARN] [10/29/2024 01:13:43.803] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower616] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@546b7af7 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower617 updated: matchIndex: -1, nextIndex: 0 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower616 updated: matchIndex: -1, nextIndex: 0 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower618 updated: matchIndex: -1, nextIndex: 0 01:13:43,803 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:43.804] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower618] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@32742d53 01:13:43,827 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:13:43,827 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:13:43,827 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:13:43,828 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower616] 01:13:43,839 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower618] 01:13:43,850 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:13:43,850 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:13:43,850 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: onRequestLeadership RequestLeadership [requestedFollowerId=follower617, replyTo=Actor[pekko://test/user/$d#-374892950]] 01:13:43,850 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Initiating leader transfer 01:13:43,850 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader615: pauseLeader successfully completed - doing transfer 01:13:43,850 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Attempting to transfer leadership 01:13:43,850 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,850 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: onLeaderTransitioning: LeaderTransitioning [leaderId=leader615] 01:13:43,850 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,850 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:43.850] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: LeaderTransitioning [leaderId=leader615] 01:13:43,851 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 01:13:43,851 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): tryToCompleteLeadershipTransfer: followerId: follower617, matchIndex: -1, lastIndex: -1, isVoting: true 01:13:43,851 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Follower's log matches - sending ElectionTimeout 01:13:43,851 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,851 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,851 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,851 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leader transfer complete 01:13:43,851 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader transfer complete - waiting for new leader [WARN] [10/29/2024 01:13:43.851] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: LeaderTransitioning [leaderId=leader615] 01:13:43,852 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 01:13:43,852 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:13:43,852 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Received TimeoutNow - switching to Candidate 01:13:43,852 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Election: Candidate has following voting peers: [follower616, leader615] [WARN] [10/29/2024 01:13:43.852] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:13:43,852 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=3, votedFor=follower617 [WARN] [10/29/2024 01:13:43.852] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:13:43,852 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Starting new election term 3 [WARN] [10/29/2024 01:13:43.852] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower617] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5b4fc4fc [WARN] [10/29/2024 01:13:43.852] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:13:43,852 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] to peer follower616 [WARN] [10/29/2024 01:13:43.852] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:13:43,852 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] to peer leader615 01:13:43,852 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:13:43,853 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:13:43,853 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Cluster state: [] 01:13:43,853 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:13:43,853 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 01:13:43,853 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=3, votedFor=null 01:13:43,853 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leadership transfer in progress - processing RequestVote 01:13:43,853 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): In requestVote: RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:13:43,853 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=3, votedFor=follower617 01:13:43,853 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:13:43,853 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader) :- Switching from behavior Leader to Follower, election term: 3 [WARN] [10/29/2024 01:13:43.853] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/follower617]: RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] 01:13:43,854 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 01:13:43,854 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 01:13:43,855 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Election: Leader has following peers: [follower616, follower618, leader615] 01:13:43,855 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,855 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/29/2024 01:13:43.855] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/follower617]: AppendEntries [leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:13:43,855 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,855 AM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [10/29/2024 01:13:43.855] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/follower617]: AppendEntries [leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:13:43,855 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - leader615: Last sync leader does not match current leader follower617, need to catch up to -1 01:13:43,855 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader changed to follower617 01:13:43,855 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader615 updated: matchIndex: -1, nextIndex: 0 01:13:43,855 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:13:43,855 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorLeadershipTransferCohort - leader615: Successfully transferred leadership to follower617 in 5.276 ms 01:13:43,855 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Leadership transferred successfully to follower617 [WARN] [10/29/2024 01:13:43.856] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/leader615] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1b6809db 01:13:43,901 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 01:13:43,901 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower616-notifier619#-1551324491] 01:13:43,912 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower616-collector620#190956729] 01:13:43,913 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower617-notifier621#-943769775] 01:13:43,913 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower617-collector622#-1729903734] 01:13:43,913 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower617] 01:13:43,913 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower618-notifier623#278059302] 01:13:43,913 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower618-collector624#-735825811] 01:13:43,913 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader615-notifier625#589296302] 01:13:43,913 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader615-collector626#-1429269389] 01:13:43,914 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader615] 01:13:43,914 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$d#-374892950] 01:13:43,919 AM [test-pekko.actor.default-dispatcher-11] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [INFO] [10/29/2024 01:13:43.919] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [10/29/2024 01:13:43.920] [test-pekko.actor.default-dispatcher-18] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.170.219:17355) 01:13:43,922 AM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.219:17355] - Exiting completed 01:13:43,922 AM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.219:17355] - Shutting down... 01:13:43,922 AM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.219:17355] - Successfully shut down [INFO] [10/29/2024 01:13:43.921] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.219:17355] - Exiting completed [INFO] [10/29/2024 01:13:43.921] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.219:17355] - Shutting down... 01:13:43,923 AM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 01:13:43,923 AM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [10/29/2024 01:13:43.921] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.219:17355] - Successfully shut down [INFO] [10/29/2024 01:13:43.923] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.170.219:17355/system/remoting-terminator] Shutting down remote daemon. [INFO] [10/29/2024 01:13:43.923] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.170.219:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 01:13:43,925 AM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [INFO] [10/29/2024 01:13:43.925] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.170.219:17355/system/remoting-terminator] Remoting shut down. [DEBUG] [10/29/2024 01:13:43.926] [test-pekko.actor.internal-dispatcher-3] [EventStream] shutting down: StandardOutLogger