[DEBUG] [08/01/2024 12:31:46.545] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 12:31:46,546 PM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [08/01/2024 12:31:46.546] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [08/01/2024 12:31:46.546] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [08/01/2024 12:31:46.554] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.103:17355] with UID [1850734388015188165] 12:31:46,555 PM [test-pekko.actor.default-dispatcher-8] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.103:17355] with UID [1850734388015188165] 12:31:46,555 PM [test-pekko.actor.default-dispatcher-8] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.103:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [08/01/2024 12:31:46.555] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.103:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [08/01/2024 12:31:46.558] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.103:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [08/01/2024 12:31:46.558] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.103:17355] - Started up successfully 12:31:46,558 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.103:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 12:31:46,558 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.103:17355] - Started up successfully [INFO] [08/01/2024 12:31:46.566] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.103:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [08/01/2024 12:31:46.566] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.103:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 12:31:46,566 PM [test-pekko.actor.default-dispatcher-8] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.103:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 12:31:46,567 PM [test-pekko.actor.default-dispatcher-8] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.103:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 12:31:46,571 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 12:31:46,571 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/01/2024 12:31:46.573] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower551]: org.opendaylight.controller.cluster.common.actor.Monitor@7a09e839 [INFO] [pekkoDeadLetter][08/01/2024 12:31:46.573] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower551] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:46,575 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower551 with journal batch size 1000 [WARN] [08/01/2024 12:31:46.577] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower552]: org.opendaylight.controller.cluster.common.actor.Monitor@3f0b791c 12:31:46,577 PM [test-pekko.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower551] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:46,578 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: SnapshotOffer called. 12:31:46,578 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Moving last applied index from -1 to -1 12:31:46,578 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=1, votedFor=null 12:31:46,578 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING]] 12:31:46,578 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower551 in 75.70 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:46,579 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Recovery completed in 752.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:46,580 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower552 with journal batch size 1000 [INFO] [pekkoDeadLetter][08/01/2024 12:31:46.580] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower552] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:46,580 PM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower552] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:46,582 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: SnapshotOffer called. 12:31:46,582 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Moving last applied index from -1 to -1 12:31:46,582 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=1, votedFor=null 12:31:46,582 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 12:31:46,582 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower552 in 74.70 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [08/01/2024 12:31:46.583] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower553]: org.opendaylight.controller.cluster.common.actor.Monitor@58abc73a [INFO] [pekkoDeadLetter][08/01/2024 12:31:46.583] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower553] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:46,583 PM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower553] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:46,583 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Recovery completed in 909.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:46,583 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower553 with journal batch size 1000 12:31:46,584 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: SnapshotOffer called. 12:31:46,584 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Moving last applied index from -1 to -1 12:31:46,584 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Set currentTerm=1, votedFor=null 12:31:46,584 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Updated server config: isVoting: false, peers: [PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 12:31:46,584 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower553 in 50.46 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:46,584 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Recovery completed in 471.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:46,685 PM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader550] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [08/01/2024 12:31:46.685] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader550]: org.opendaylight.controller.cluster.common.actor.Monitor@777d06bb [INFO] [pekkoDeadLetter][08/01/2024 12:31:46.685] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader550] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:46,685 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader550 with journal batch size 1000 12:31:46,686 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: SnapshotOffer called. 12:31:46,686 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Moving last applied index from -1 to -1 12:31:46,686 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Set currentTerm=1, votedFor=null 12:31:46,686 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 12:31:46,686 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader550 in 95.51 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:46,687 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Recovery completed in 1.485 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:47,56 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:31:47,56 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.103:17355, Removed) 12:31:47,56 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Received ElectionTimeout - switching to Candidate 12:31:47,56 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Election: Candidate has following voting peers: [follower552, follower551] 12:31:47,57 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Set currentTerm=2, votedFor=leader550 12:31:47,57 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Starting new election term 2 12:31:47,57 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] to peer follower552 12:31:47,57 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] to peer follower551 12:31:47,57 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:31:47,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:47,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Cluster state: [] 12:31:47,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:47,57 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:47,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=2, votedFor=null 12:31:47,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): In requestVote: RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:47,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=2, votedFor=leader550 12:31:47,57 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:47,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:47,58 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Cluster state: [] 12:31:47,58 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:47,58 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:47,58 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=2, votedFor=null 12:31:47,58 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): In requestVote: RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:47,58 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=2, votedFor=leader550 12:31:47,58 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:47,59 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:31:47,59 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:31:47,59 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Election: Leader has following peers: [follower553, follower552, follower551] 12:31:47,59 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,59 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,59 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,59 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower552: Last sync leader does not match current leader leader550, need to catch up to -1 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553 (Follower): Term 2 in "AppendEntries [leaderId=leader550, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Set currentTerm=2, votedFor=null 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower553: Last sync leader does not match current leader leader550, need to catch up to -1 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower551: Last sync leader does not match current leader leader550, need to catch up to -1 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower552 updated: matchIndex: -1, nextIndex: 0 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower553 updated: matchIndex: -1, nextIndex: 0 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower551 updated: matchIndex: -1, nextIndex: 0 12:31:47,60 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:47.060] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower552] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@237ca615 [WARN] [08/01/2024 12:31:47.060] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower551] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@57d7d84c [INFO] [pekkoDeadLetter][08/01/2024 12:31:47.061] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][08/01/2024 12:31:47.061] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:47,61 PM [test-pekko.actor.default-dispatcher-5] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:47,61 PM [test-pekko.actor.default-dispatcher-5] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [08/01/2024 12:31:47.061] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower553] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@18e38cb [INFO] [pekkoDeadLetter][08/01/2024 12:31:47.061] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:47,61 PM [test-pekko.actor.default-dispatcher-16] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:47,87 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:31:47,87 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader550 starting 12:31:47,89 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: onShutDown 12:31:47,89 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Initiating leader transfer 12:31:47,90 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader550: pauseLeader successfully completed - doing transfer 12:31:47,90 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Attempting to transfer leadership 12:31:47,90 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 12:31:47,90 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,90 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,90 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 12:31:47,90 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,90 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 12:31:47,90 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 12:31:47,90 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 12:31:47,90 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [08/01/2024 12:31:47.090] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower553] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@789fafb1 [WARN] [08/01/2024 12:31:47.090] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower551] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@57afd2e4 [INFO] [pekkoDeadLetter][08/01/2024 12:31:47.090] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][08/01/2024 12:31:47.090] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:47,90 PM [test-pekko.actor.default-dispatcher-16] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:47,90 PM [test-pekko.actor.default-dispatcher-16] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [08/01/2024 12:31:47.090] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower552] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5f5ca889 12:31:47,90 PM [test-pekko.actor.default-dispatcher-16] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][08/01/2024 12:31:47.090] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 12:31:47,177 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 12:31:47,177 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 12:31:47,177 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 12:31:47,297 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 12:31:47,297 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 12:31:47,297 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 12:31:47,417 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader550: leader transfer aborted 12:31:47,417 PM [test-pekko.actor.default-dispatcher-6] [WARN] RaftActorLeadershipTransferCohort - leader550: Failed to transfer leadership in 327.2 ms 12:31:47,417 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: leader transfer failed - sending PoisonPill 12:31:47,417 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Leadership transfer expired 12:31:47,418 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [08/01/2024 12:31:47.418] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/leader550] received dead letter from TestActor[pekko://test/user/follower553]: AppendEntriesReply [term=2, success=true, followerId=follower553, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:31:47,418 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [08/01/2024 12:31:47.418] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/leader550] received dead letter from TestActor[pekko://test/user/follower552]: AppendEntriesReply [term=2, success=true, followerId=follower552, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:31:47,418 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [08/01/2024 12:31:47.418] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader550] received dead letter from TestActor[pekko://test/user/follower551]: AppendEntriesReply [term=2, success=true, followerId=follower551, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:31:47,419 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader550 ending 12:31:47,421 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 12:31:47,421 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower551-notifier554#-1941700964] 12:31:47,433 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower551-collector555#1430559129] 12:31:47,433 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower551] 12:31:47,433 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower552-notifier556#-1908916079] 12:31:47,433 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower552-collector557#-1951676156] 12:31:47,433 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower552] 12:31:47,433 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower553-notifier558#1460726971] 12:31:47,433 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower553-collector559#1192721390] 12:31:47,433 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower553] 12:31:47,433 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader550-notifier560#-375184790] 12:31:47,434 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader550-collector561#-328091365] 12:31:47,434 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader550] 12:31:47,435 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 12:31:47,435 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/01/2024 12:31:47.436] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower563]: org.opendaylight.controller.cluster.common.actor.Monitor@16b59275 12:31:47,436 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower563 with journal batch size 1000 12:31:47,437 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: SnapshotOffer called. 12:31:47,437 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Moving last applied index from -1 to -1 12:31:47,437 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=1, votedFor=null 12:31:47,437 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 12:31:47,437 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower563 in 59.81 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:47,438 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Recovery completed in 817.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:47,440 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower564 with journal batch size 1000 [WARN] [08/01/2024 12:31:47.440] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower564]: org.opendaylight.controller.cluster.common.actor.Monitor@79ab5fcf 12:31:47,440 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: SnapshotOffer called. 12:31:47,440 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Moving last applied index from -1 to -1 12:31:47,440 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=1, votedFor=null 12:31:47,440 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 12:31:47,440 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower564 in 79.46 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:47,442 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Recovery completed in 1.457 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/01/2024 12:31:47.442] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower565]: org.opendaylight.controller.cluster.common.actor.Monitor@5b6337ba 12:31:47,443 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower565 with journal batch size 1000 12:31:47,443 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: SnapshotOffer called. 12:31:47,443 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Moving last applied index from -1 to -1 12:31:47,443 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=1, votedFor=null 12:31:47,443 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Updated server config: isVoting: false, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING]] 12:31:47,443 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower565 in 81.92 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:47,444 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Recovery completed in 616.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/01/2024 12:31:47.445] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader562]: org.opendaylight.controller.cluster.common.actor.Monitor@331a18ea 12:31:47,445 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader562 with journal batch size 1000 12:31:47,445 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: SnapshotOffer called. 12:31:47,445 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Moving last applied index from -1 to -1 12:31:47,445 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=1, votedFor=null 12:31:47,445 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Updated server config: isVoting: true, peers: [PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 12:31:47,445 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader562 in 39.74 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:47,446 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Recovery completed in 400.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:47,826 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:31:47,826 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.103:17355, Removed) 12:31:47,826 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Received ElectionTimeout - switching to Candidate 12:31:47,826 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Election: Candidate has following voting peers: [follower564, follower563] 12:31:47,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=2, votedFor=leader562 12:31:47,827 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Starting new election term 2 12:31:47,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] to peer follower564 12:31:47,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] to peer follower563 12:31:47,827 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:47,827 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Cluster state: [] 12:31:47,827 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:47,827 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:47,827 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:47,827 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 12:31:47,827 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=2, votedFor=null 12:31:47,827 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:47,827 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:47,827 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=2, votedFor=null 12:31:47,827 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): In requestVote: RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:47,827 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:47,827 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=2, votedFor=leader562 12:31:47,827 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=2, votedFor=leader562 12:31:47,827 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:47,827 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:47,829 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:31:47,829 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:31:47,829 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:31:47,829 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Election: Leader has following peers: [follower564, follower563, follower565] 12:31:47,830 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,830 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,830 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower564: Last sync leader does not match current leader leader562, need to catch up to -1 12:31:47,830 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader leader562, need to catch up to -1 12:31:47,830 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:47.830] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4063f9e1 12:31:47,830 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 [WARN] [08/01/2024 12:31:47.830] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4409231b 12:31:47,830 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 2 in "AppendEntries [leaderId=leader562, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:31:47,830 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=2, votedFor=null 12:31:47,830 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader leader562, need to catch up to -1 12:31:47,831 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower564 updated: matchIndex: -1, nextIndex: 0 12:31:47,831 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,831 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 12:31:47,831 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,831 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 12:31:47,831 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:47.831] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7ceb0160 12:31:47,850 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:31:47,851 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 12:31:47,851 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 12:31:47,852 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 12:31:47,852 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: onRequestLeadership RequestLeadership [requestedFollowerId=follower564, replyTo=Actor[pekko://test/user/$a#-676771333]] 12:31:47,854 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Initiating leader transfer 12:31:47,854 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader562: pauseLeader successfully completed - doing transfer 12:31:47,854 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Attempting to transfer leadership 12:31:47,854 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,854 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,854 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,854 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): tryToCompleteLeadershipTransfer: followerId: follower564, matchIndex: -1, lastIndex: -1, isVoting: true 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Follower's log matches - sending ElectionTimeout 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leader transfer complete 12:31:47,855 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader transfer complete - waiting for new leader [WARN] [08/01/2024 12:31:47.855] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5d0bfc0e [WARN] [08/01/2024 12:31:47.856] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6cb79e86 [WARN] [08/01/2024 12:31:47.856] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@66366896 12:31:47,856 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 12:31:47,856 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:31:47,856 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Received TimeoutNow - switching to Candidate 12:31:47,856 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Election: Candidate has following voting peers: [leader562, follower563] 12:31:47,856 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=3, votedFor=follower564 12:31:47,856 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Starting new election term 3 12:31:47,856 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] to peer leader562 12:31:47,856 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] to peer follower563 12:31:47,856 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 2 - updating term 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=3, votedFor=null 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=3, votedFor=follower564 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Cluster state: [] 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=3, votedFor=null 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leadership transfer in progress - processing RequestVote 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): In requestVote: RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=3, votedFor=follower564 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader) :- Switching from behavior Leader to Follower, election term: 3 12:31:47,857 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Election: Leader has following peers: [leader562, follower563, follower565] 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,858 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - leader562: Last sync leader does not match current leader follower564, need to catch up to -1 12:31:47,858 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader follower564, need to catch up to -1 12:31:47,858 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader changed to follower564 12:31:47,858 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 3 in "AppendEntries [leaderId=follower564, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 12:31:47,858 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=3, votedFor=null 12:31:47,858 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader follower564, need to catch up to -1 12:31:47,858 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorLeadershipTransferCohort - leader562: Successfully transferred leadership to follower564 in 4.146 ms [WARN] [08/01/2024 12:31:47.858] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4db58cb4 12:31:47,858 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Leadership transferred successfully to follower564 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader562 updated: matchIndex: -1, nextIndex: 0 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,858 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 12:31:47,859 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:47,859 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending [WARN] [08/01/2024 12:31:47.859] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/leader562] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3a615ce6 [WARN] [08/01/2024 12:31:47.859] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@12bf9b86 12:31:47,859 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 12:31:47,859 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower563-notifier566#824522574] 12:31:47,860 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower563-collector567#1718184539] 12:31:47,860 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower563] 12:31:47,860 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower564-notifier568#-1445891221] 12:31:47,860 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower564-collector569#1178706929] 12:31:47,860 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower564] 12:31:47,861 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower565-notifier570#-1478240208] 12:31:47,861 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower565-collector571#-963149999] 12:31:47,861 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower565] 12:31:47,861 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader562-notifier572#-1719547507] 12:31:47,861 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader562-collector573#-1253395239] 12:31:47,861 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader562] 12:31:47,861 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$a#-676771333] 12:31:47,862 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 12:31:47,862 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/01/2024 12:31:47.863] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower575]: org.opendaylight.controller.cluster.common.actor.Monitor@6533b9d8 12:31:47,864 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower575 with journal batch size 1000 12:31:47,864 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: SnapshotOffer called. 12:31:47,864 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Moving last applied index from -1 to -1 12:31:47,864 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=1, votedFor=null 12:31:47,864 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 12:31:47,864 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower575 in 51.45 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:47,864 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Recovery completed in 530.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/01/2024 12:31:47.865] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower576]: org.opendaylight.controller.cluster.common.actor.Monitor@2eedfde0 12:31:47,866 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower576 with journal batch size 1000 12:31:47,866 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: SnapshotOffer called. 12:31:47,866 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Moving last applied index from -1 to -1 12:31:47,866 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=1, votedFor=null 12:31:47,866 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING]] 12:31:47,866 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower576 in 57.64 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:47,867 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Recovery completed in 589.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/01/2024 12:31:47.869] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower577]: org.opendaylight.controller.cluster.common.actor.Monitor@24f21a84 12:31:47,870 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower577 with journal batch size 1000 12:31:47,870 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: SnapshotOffer called. 12:31:47,870 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Moving last applied index from -1 to -1 12:31:47,870 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Set currentTerm=1, votedFor=null 12:31:47,870 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Updated server config: isVoting: false, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 12:31:47,870 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower577 in 74.64 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [08/01/2024 12:31:47.871] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader574]: org.opendaylight.controller.cluster.common.actor.Monitor@36d6c0e9 12:31:47,871 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Recovery completed in 742.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:47,871 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader574 with journal batch size 1000 12:31:47,872 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: SnapshotOffer called. 12:31:47,872 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Moving last applied index from -1 to -1 12:31:47,872 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Set currentTerm=1, votedFor=null 12:31:47,872 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Updated server config: isVoting: true, peers: [PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 12:31:47,872 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader574 in 40.40 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:47,873 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Recovery completed in 1.182 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:48,217 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:31:48,217 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.103:17355, Removed) 12:31:48,217 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Received ElectionTimeout - switching to Candidate 12:31:48,217 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Election: Candidate has following voting peers: [follower575, follower576] 12:31:48,217 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Set currentTerm=2, votedFor=leader574 12:31:48,217 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Starting new election term 2 12:31:48,217 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] to peer follower575 12:31:48,217 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] to peer follower576 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Cluster state: [] 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=2, votedFor=null 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): In requestVote: RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=2, votedFor=leader574 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:48,218 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:48,218 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Cluster state: [] 12:31:48,218 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:48,218 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:48,218 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=2, votedFor=null 12:31:48,218 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): In requestVote: RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:48,218 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=2, votedFor=leader574 12:31:48,218 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:31:48,218 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Election: Leader has following peers: [follower575, follower577, follower576] 12:31:48,219 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,219 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,219 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,219 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577 (Follower): Term 2 in "AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:31:48,219 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Set currentTerm=2, votedFor=null 12:31:48,219 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower577: Last sync leader does not match current leader leader574, need to catch up to -1 12:31:48,219 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower575: Last sync leader does not match current leader leader574, need to catch up to -1 [WARN] [08/01/2024 12:31:48.219] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower575] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2886f85f 12:31:48,219 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:31:48,219 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower576: Last sync leader does not match current leader leader574, need to catch up to -1 [WARN] [08/01/2024 12:31:48.219] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower577] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@113533c2 [WARN] [08/01/2024 12:31:48.219] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower576] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@df8fc81 12:31:48,220 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower577 updated: matchIndex: -1, nextIndex: 0 12:31:48,220 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,220 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower575 updated: matchIndex: -1, nextIndex: 0 12:31:48,220 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,220 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower576 updated: matchIndex: -1, nextIndex: 0 12:31:48,220 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,225 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:31:48,225 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 12:31:48,226 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 12:31:48,226 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower576 starting 12:31:48,226 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: onShutDown 12:31:48,226 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower576 ending 12:31:48,226 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 12:31:48,226 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 12:31:48,226 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: onRequestLeadership RequestLeadership [requestedFollowerId=follower576, replyTo=Actor[pekko://test/user/$b#-1391717654]] 12:31:48,226 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Initiating leader transfer 12:31:48,227 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] 12:31:48,227 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] 12:31:48,227 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader574: pauseLeader successfully completed - doing transfer 12:31:48,227 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Attempting to transfer leadership [WARN] [08/01/2024 12:31:48.227] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: LeaderTransitioning [leaderId=leader574] 12:31:48,227 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,227 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,227 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 12:31:48,227 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [08/01/2024 12:31:48.227] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower577] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3b6bc0f5 [WARN] [08/01/2024 12:31:48.227] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower575] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@56de15d7 12:31:48,227 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:48.227] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:31:48,337 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 12:31:48,337 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [08/01/2024 12:31:48.337] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:31:48,456 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 12:31:48,457 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [08/01/2024 12:31:48.457] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:31:48,576 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader574: leader transfer aborted 12:31:48,576 PM [test-pekko.actor.default-dispatcher-15] [WARN] RaftActorLeadershipTransferCohort - leader574: Failed to transfer leadership in 349.7 ms 12:31:48,576 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: LeadershipTransfer request from follower576 failed 12:31:48,577 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Leadership transfer expired 12:31:48,578 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 12:31:48,578 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [08/01/2024 12:31:48.578] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:31:48,578 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 12:31:48,579 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower575-notifier578#834185619] 12:31:48,580 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower575-collector579#1816836462] 12:31:48,580 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower575] 12:31:48,580 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower576-notifier580#-863232911] 12:31:48,580 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower576-collector581#-1368915665] 12:31:48,580 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower576] 12:31:48,581 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower577-notifier582#1057485661] 12:31:48,581 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower577-collector583#729751973] 12:31:48,581 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower577] 12:31:48,581 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader574-notifier584#2136077546] 12:31:48,581 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader574-collector585#-152330455] 12:31:48,581 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader574] 12:31:48,581 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$b#-1391717654] 12:31:48,583 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 12:31:48,583 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/01/2024 12:31:48.585] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower587]: org.opendaylight.controller.cluster.common.actor.Monitor@1c8727e2 12:31:48,585 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower587 with journal batch size 1000 12:31:48,586 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: SnapshotOffer called. 12:31:48,586 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to -1 12:31:48,586 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=1, votedFor=null 12:31:48,586 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING]] 12:31:48,586 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower587 in 86.53 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:48,587 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Recovery completed in 1.001 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/01/2024 12:31:48.588] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower588]: org.opendaylight.controller.cluster.common.actor.Monitor@5577c08c 12:31:48,589 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower588 with journal batch size 1000 12:31:48,589 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: SnapshotOffer called. 12:31:48,589 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to -1 12:31:48,589 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=1, votedFor=null 12:31:48,589 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 12:31:48,589 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower588 in 73.09 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [08/01/2024 12:31:48.590] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower589]: org.opendaylight.controller.cluster.common.actor.Monitor@37d7dbe5 12:31:48,591 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower589 with journal batch size 1000 12:31:48,591 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: SnapshotOffer called. 12:31:48,591 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to -1 12:31:48,591 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=1, votedFor=null 12:31:48,591 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Updated server config: isVoting: false, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 12:31:48,591 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower589 in 49.70 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:48,591 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Recovery completed in 1.910 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:48,591 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Recovery completed in 502.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:48,593 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader586 with journal batch size 1000 [WARN] [08/01/2024 12:31:48.593] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader586]: org.opendaylight.controller.cluster.common.actor.Monitor@5ea3ccc4 12:31:48,593 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: SnapshotOffer called. 12:31:48,593 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to -1 12:31:48,593 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=1, votedFor=null 12:31:48,593 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Updated server config: isVoting: true, peers: [PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 12:31:48,593 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader586 in 191.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:48,594 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Recovery completed in 638.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:48,937 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:31:48,937 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.103:17355, Removed) 12:31:48,937 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Received ElectionTimeout - switching to Candidate 12:31:48,937 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Election: Candidate has following voting peers: [follower588, follower587] 12:31:48,937 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=2, votedFor=leader586 12:31:48,937 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Starting new election term 2 12:31:48,937 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] to peer follower588 12:31:48,937 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] to peer follower587 12:31:48,937 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:31:48,937 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:48,937 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 12:31:48,937 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:48,938 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:48,938 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=2, votedFor=null 12:31:48,938 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:48,938 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Cluster state: [] 12:31:48,938 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:48,938 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:48,938 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=2, votedFor=null 12:31:48,938 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:48,938 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=2, votedFor=leader586 12:31:48,938 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:48,938 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): In requestVote: RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:48,938 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=2, votedFor=leader586 12:31:48,938 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:48,938 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:31:48,938 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:31:48,938 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Election: Leader has following peers: [follower589, follower588, follower587] 12:31:48,938 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,939 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,939 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,939 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:31:48,939 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 2 in "AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:31:48,939 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=2, votedFor=null 12:31:48,939 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower587: Last sync leader does not match current leader leader586, need to catch up to -1 12:31:48,939 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader leader586, need to catch up to -1 12:31:48,939 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: -1, nextIndex: 0 12:31:48,939 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader leader586, need to catch up to -1 12:31:48,939 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:48,939 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: -1, nextIndex: 0 12:31:48,939 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:48.939] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7a3afc28 12:31:48,939 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: -1, nextIndex: 0 [WARN] [08/01/2024 12:31:48.939] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7a4d78c6 12:31:48,939 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:48.940] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@773fa64d 12:31:48,948 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:31:48,948 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower589 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower589: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower587 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower587: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Number of entries to be appended = 1 12:31:48,948 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 12:31:48,948 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Append entry to log zero 12:31:48,948 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Number of entries to be appended = 1 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Append entry to log zero 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Log size is now 1 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Log size is now 1 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower587, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower589, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 12:31:48,948 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: 0, nextIndex: 1 12:31:48,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:48,949 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 12:31:48,949 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:48,949 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:48,949 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 12:31:48,949 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Setting last applied to 0 12:31:48,949 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to 0 12:31:48,949 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Applying state for log index 0 data zero 12:31:48,949 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: applyState called: zero 12:31:48,949 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:48,949 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persisting ApplyJournalEntries with index=0 [WARN] [08/01/2024 12:31:48.949] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@17158550 12:31:48,949 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [08/01/2024 12:31:48.949] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7e4176b4 12:31:49,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 12:31:49,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:49,57 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Commit index set to 0 12:31:49,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Commit index set to 0 12:31:49,57 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 12:31:49,57 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 12:31:49,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 12:31:49,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 12:31:49,57 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Setting last applied to 0 12:31:49,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Setting last applied to 0 12:31:49,57 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to 0 12:31:49,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to 0 12:31:49,57 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Applying state for log index 0 data zero 12:31:49,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Applying state for log index 0 data zero 12:31:49,57 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: applyState called: zero 12:31:49,57 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: applyState called: zero 12:31:49,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Persisting ApplyJournalEntries with index=0 12:31:49,57 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Persisting ApplyJournalEntries with index=0 12:31:49,99 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 12:31:49,99 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 12:31:49,99 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: onShutDown 12:31:49,99 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Initiating leader transfer 12:31:49,100 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 12:31:49,100 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader586: pauseLeader successfully completed - doing transfer 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Attempting to transfer leadership 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,100 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower589, matchIndex: 0, lastIndex: 0, isVoting: false 12:31:49,100 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower587, matchIndex: 0, lastIndex: 0, isVoting: true 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Follower's log matches - sending ElectionTimeout 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,100 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leader transfer complete 12:31:49,100 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader transfer complete - waiting for new leader 12:31:49,101 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 12:31:49,101 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:31:49,101 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Received TimeoutNow - switching to Candidate 12:31:49,101 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Election: Candidate has following voting peers: [leader586, follower588] 12:31:49,101 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=3, votedFor=follower587 12:31:49,101 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Starting new election term 3 12:31:49,101 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] to peer leader586 12:31:49,101 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] to peer follower588 12:31:49,101 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:49,101 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 12:31:49,101 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:49,101 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Cluster state: [] 12:31:49,101 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 12:31:49,101 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 12:31:49,101 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2]" message is greater than leader's term 2 - switching to Follower 12:31:49,101 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 12:31:49,101 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=3, votedFor=null 12:31:49,101 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2]" message is greater than follower's term 2 - updating term 12:31:49,101 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=3, votedFor=null 12:31:49,102 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leadership transfer in progress - processing RequestVote 12:31:49,102 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): In requestVote: RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 12:31:49,102 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:49,102 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=3, votedFor=follower587 12:31:49,102 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=3, votedFor=follower587 12:31:49,102 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 12:31:49,102 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 12:31:49,102 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader) :- Switching from behavior Leader to Follower, election term: 3 12:31:49,103 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 12:31:49,103 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 12:31:49,103 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Election: Leader has following peers: [leader586, follower589, follower588] 12:31:49,103 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,103 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,103 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - leader586: Last sync leader does not match current leader follower587, need to catch up to -1 12:31:49,103 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader changed to follower587 12:31:49,103 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,103 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 3 in "AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 12:31:49,104 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=3, votedFor=null 12:31:49,104 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 12:31:49,103 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorLeadershipTransferCohort - leader586: Successfully transferred leadership to follower587 in 4.068 ms 12:31:49,104 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader follower587, need to catch up to -1 12:31:49,104 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: leader transfer succeeded - sending PoisonPill 12:31:49,104 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader586 updated: matchIndex: 0, nextIndex: 1 12:31:49,104 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,104 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 12:31:49,104 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,104 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) [WARN] [08/01/2024 12:31:49.104] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/leader586] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@39ef8c0c [WARN] [08/01/2024 12:31:49.104] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@118105af [WARN] [08/01/2024 12:31:49.104] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/leader586] received dead letter from TestActor[pekko://test/user/follower587]: AppendEntries [leaderId=follower587, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [08/01/2024 12:31:49.104] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@27b48ab3 [WARN] [08/01/2024 12:31:49.617] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/leader586] received dead letter from TestActor[pekko://test/user/follower587]: AppendEntries [leaderId=follower587, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:31:49,617 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 12:31:49,617 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): sendAppendEntries: 0 is present for follower follower588 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader follower587, need to catch up to -1 12:31:49,617 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries: AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Number of entries to be appended = 1 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Append entry to log zero 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Log size is now 1 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Commit index set to 0 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries returning : AppendEntriesReply [term=3, success=true, followerId=follower588, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Setting last applied to 0 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to 0 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Applying state for log index 0 data zero 12:31:49,617 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: 0, nextIndex: 1 12:31:49,617 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:49,617 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: applyState called: zero 12:31:49,618 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:49,618 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Persisting ApplyJournalEntries with index=0 [WARN] [08/01/2024 12:31:49.619] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7d2cbae5 [WARN] [08/01/2024 12:31:49.619] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@559bef12 12:31:49,656 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 12:31:49,656 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 12:31:49,656 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower588 starting 12:31:49,657 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onShutDown 12:31:49,657 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower588 ending 12:31:49,657 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 12:31:49,657 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587-notifier590#661735085] 12:31:49,668 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587-collector591#-665411536] 12:31:49,668 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower587] 12:31:49,668 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower588-notifier592#-1884747348] 12:31:49,668 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower588-collector593#99638566] 12:31:49,668 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower588] 12:31:49,669 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower589-notifier594#1159949518] 12:31:49,669 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower589-collector595#1488868435] 12:31:49,669 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower589] 12:31:49,669 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader586-notifier596#-820609297] 12:31:49,669 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader586-collector597#-1029917622] 12:31:49,669 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader586] 12:31:49,670 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting 12:31:49,671 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader598 with journal batch size 1000 [WARN] [08/01/2024 12:31:49.672] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader598]: org.opendaylight.controller.cluster.common.actor.Monitor@4cf68c4e 12:31:49,673 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:49,673 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:31:49,673 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower): Received TimeoutNow - switching to Candidate 12:31:49,673 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Election: Candidate has following voting peers: [] 12:31:49,673 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Set currentTerm=1, votedFor=leader598 12:31:49,673 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Starting new election term 1 12:31:49,673 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 12:31:49,674 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Received ElectionTimeout 12:31:49,674 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Leader): Election: Leader has following peers: [] 12:31:49,674 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 12:31:49,674 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader598 starting 12:31:49,674 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: onShutDown 12:31:49,674 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader598 ending 12:31:49,674 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 12:31:49,674 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader598-collector602#-967530502] 12:31:49,686 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader598] 12:31:49,687 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 12:31:49,687 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:31:49,690 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower604 with journal batch size 1000 [WARN] [08/01/2024 12:31:49.690] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower604]: org.opendaylight.controller.cluster.common.actor.Monitor@7f13e2b3 12:31:49,691 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: SnapshotOffer called. 12:31:49,691 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to -1 12:31:49,691 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=1, votedFor=null 12:31:49,691 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 12:31:49,691 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower604 in 77.48 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:49,692 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Recovery completed in 782.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:49,692 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower605 with journal batch size 1000 [WARN] [08/01/2024 12:31:49.692] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower605]: org.opendaylight.controller.cluster.common.actor.Monitor@267de0ac 12:31:49,693 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: SnapshotOffer called. 12:31:49,693 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Moving last applied index from -1 to -1 12:31:49,693 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=1, votedFor=null 12:31:49,693 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Updated server config: isVoting: true, peers: [PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 12:31:49,693 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower605 in 62.67 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:49,694 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Recovery completed in 595.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/01/2024 12:31:49.694] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower606]: org.opendaylight.controller.cluster.common.actor.Monitor@16f78f54 12:31:49,695 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower606 with journal batch size 1000 12:31:49,696 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader603 with journal batch size 1000 [WARN] [08/01/2024 12:31:49.696] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader603]: org.opendaylight.controller.cluster.common.actor.Monitor@13eae618 12:31:49,697 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: SnapshotOffer called. 12:31:49,697 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to -1 12:31:49,697 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Set currentTerm=1, votedFor=null 12:31:49,698 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Updated server config: isVoting: false, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 12:31:49,698 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: SnapshotOffer called. 12:31:49,698 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to -1 12:31:49,698 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Set currentTerm=1, votedFor=null 12:31:49,698 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower606 in 189.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:49,698 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING]] 12:31:49,698 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader603 in 61.85 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:49,699 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Recovery completed in 1.012 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:49,699 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Recovery completed in 700.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:50,117 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:31:50,117 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.103:17355, Removed) 12:31:50,117 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Received ElectionTimeout - switching to Candidate 12:31:50,117 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Election: Candidate has following voting peers: [follower605, follower604] 12:31:50,117 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Set currentTerm=2, votedFor=leader603 12:31:50,117 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Starting new election term 2 12:31:50,117 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] to peer follower605 12:31:50,117 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] to peer follower604 12:31:50,118 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:31:50,118 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:50,118 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:50,118 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Cluster state: [] 12:31:50,118 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:50,118 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Cluster state: [] 12:31:50,118 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:50,118 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:50,118 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=2, votedFor=null 12:31:50,118 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:50,118 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=2, votedFor=null 12:31:50,118 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): In requestVote: RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:50,118 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=2, votedFor=leader603 12:31:50,118 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): In requestVote: RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:50,118 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=2, votedFor=leader603 12:31:50,118 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:50,118 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:50,119 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:31:50,119 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:31:50,119 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Election: Leader has following peers: [follower605, follower604, follower606] 12:31:50,119 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:50,119 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:50,119 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:50,119 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:31:50,119 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Term 2 in "AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:31:50,119 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Set currentTerm=2, votedFor=null 12:31:50,119 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower606: Last sync leader does not match current leader leader603, need to catch up to -1 12:31:50,120 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: -1, nextIndex: 0 12:31:50,120 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:50,120 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower605: Last sync leader does not match current leader leader603, need to catch up to -1 12:31:50,120 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower605 updated: matchIndex: -1, nextIndex: 0 12:31:50,120 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:50.120] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower605] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@58c5338e 12:31:50,120 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] SyncStatusTracker - follower604: Last sync leader does not match current leader leader603, need to catch up to -1 [WARN] [08/01/2024 12:31:50.120] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower606] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6fe6481a [WARN] [08/01/2024 12:31:50.120] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/follower604] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2334e37c 12:31:50,120 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: -1, nextIndex: 0 12:31:50,121 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:50,153 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:31:50,153 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 12:31:50,154 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 12:31:50,154 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower604 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower604: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower606 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower606: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Number of entries to be appended = 1 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 12:31:50,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Append entry to log zero 12:31:50,155 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:50,154 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:50,155 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Number of entries to be appended = 1 12:31:50,155 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Log size is now 1 12:31:50,155 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 12:31:50,155 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Append entry to log zero 12:31:50,155 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:50,155 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower606, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:31:50,155 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 12:31:50,155 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Log size is now 1 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: 0, nextIndex: 1 12:31:50,155 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower604, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 12:31:50,155 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: 0, nextIndex: 1 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 12:31:50,155 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Setting last applied to 0 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to 0 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Applying state for log index 0 data zero 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: applyState called: zero 12:31:50,155 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [08/01/2024 12:31:50.155] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower606] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2056307c 12:31:50,155 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persisting ApplyJournalEntries with index=0 [WARN] [08/01/2024 12:31:50.156] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower604] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3c39c27 12:31:50,237 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 12:31:50,237 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:50,357 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 12:31:50,357 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 12:31:50,357 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Commit index set to 0 12:31:50,357 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 12:31:50,357 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 12:31:50,357 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Setting last applied to 0 12:31:50,357 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to 0 12:31:50,357 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Applying state for log index 0 data zero 12:31:50,357 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: applyState called: zero 12:31:50,358 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Persisting ApplyJournalEntries with index=0 12:31:50,358 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Commit index set to 0 12:31:50,358 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 12:31:50,358 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 12:31:50,358 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Setting last applied to 0 12:31:50,358 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to 0 12:31:50,358 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Applying state for log index 0 data zero 12:31:50,358 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: applyState called: zero 12:31:50,358 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Persisting ApplyJournalEntries with index=0 12:31:50,406 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 12:31:50,406 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 12:31:50,406 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 12:31:50,406 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: onRequestLeadership RequestLeadership [requestedFollowerId=follower605, replyTo=Actor[pekko://test/user/$c#-1577245314]] 12:31:50,406 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Initiating leader transfer 12:31:50,407 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 12:31:50,407 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 12:31:50,407 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader603: pauseLeader successfully completed - doing transfer 12:31:50,407 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Attempting to transfer leadership 12:31:50,407 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 12:31:50,407 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:50,407 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 12:31:50,407 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:50,407 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 12:31:50,407 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 12:31:50,407 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 12:31:50,597 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 12:31:50,597 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 12:31:50,717 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] RaftActorLeadershipTransferCohort - leader603: leader transfer aborted 12:31:50,717 PM [test-pekko.actor.default-dispatcher-8] [WARN] RaftActorLeadershipTransferCohort - leader603: Failed to transfer leadership in 310.6 ms 12:31:50,717 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: LeadershipTransfer request from follower605 failed 12:31:50,717 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Leadership transfer expired 12:31:50,717 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 12:31:50,718 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 12:31:50,759 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 12:31:50,759 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower604-notifier607#-1239832951] 12:31:50,770 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower604-collector608#507983214] 12:31:50,770 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower604] 12:31:50,771 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower605-notifier609#-1515369828] 12:31:50,771 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower605-collector610#-1484526968] 12:31:50,771 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower605] 12:31:50,771 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower606-notifier611#-988374723] 12:31:50,771 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower606-collector612#-1680216439] 12:31:50,771 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower606] 12:31:50,772 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader603-notifier613#-1466571692] 12:31:50,772 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader603-collector614#-543967860] 12:31:50,772 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader603] 12:31:50,772 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$c#-1577245314] 12:31:50,773 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 12:31:50,773 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/01/2024 12:31:50.776] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower616]: org.opendaylight.controller.cluster.common.actor.Monitor@1f464dfe 12:31:50,776 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower616 with journal batch size 1000 12:31:50,777 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: SnapshotOffer called. 12:31:50,777 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Moving last applied index from -1 to -1 12:31:50,777 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=1, votedFor=null 12:31:50,777 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Updated server config: isVoting: true, peers: [PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 12:31:50,777 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower616 in 67.35 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:50,778 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Recovery completed in 714.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/01/2024 12:31:50.780] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower617]: org.opendaylight.controller.cluster.common.actor.Monitor@313042e 12:31:50,781 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower617 with journal batch size 1000 12:31:50,782 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: SnapshotOffer called. 12:31:50,782 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Moving last applied index from -1 to -1 12:31:50,782 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=1, votedFor=null 12:31:50,782 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 12:31:50,782 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower617 in 106.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [08/01/2024 12:31:50.783] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower618]: org.opendaylight.controller.cluster.common.actor.Monitor@6941e365 12:31:50,783 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Recovery completed in 1.028 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:50,783 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower618 with journal batch size 1000 12:31:50,783 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: SnapshotOffer called. 12:31:50,783 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Moving last applied index from -1 to -1 12:31:50,783 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Set currentTerm=1, votedFor=null 12:31:50,783 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Updated server config: isVoting: false, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 12:31:50,783 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower618 in 119.6 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:50,784 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Recovery completed in 514.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/01/2024 12:31:50.885] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader615]: org.opendaylight.controller.cluster.common.actor.Monitor@3621cfea 12:31:50,886 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader615 with journal batch size 1000 12:31:50,886 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: SnapshotOffer called. 12:31:50,886 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Moving last applied index from -1 to -1 12:31:50,886 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=1, votedFor=null 12:31:50,886 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING]] 12:31:50,886 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader615 in 104.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 12:31:50,894 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Recovery completed in 7.702 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 12:31:51,256 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:31:51,256 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.103:17355, Removed) 12:31:51,256 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Received ElectionTimeout - switching to Candidate 12:31:51,256 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Election: Candidate has following voting peers: [follower616, follower617] 12:31:51,256 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=2, votedFor=leader615 12:31:51,257 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Starting new election term 2 12:31:51,257 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] to peer follower616 12:31:51,257 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] to peer follower617 12:31:51,257 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:51,257 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:51,257 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Cluster state: [] 12:31:51,257 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:51,257 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Cluster state: [] 12:31:51,257 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:51,257 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=2, votedFor=null 12:31:51,257 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:51,257 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 12:31:51,258 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=2, votedFor=null 12:31:51,258 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): In requestVote: RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:51,258 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): In requestVote: RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:51,258 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=2, votedFor=leader615 12:31:51,258 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=2, votedFor=leader615 12:31:51,258 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:51,258 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 12:31:51,258 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 12:31:51,259 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 12:31:51,259 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 12:31:51,259 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Election: Leader has following peers: [follower616, follower618, follower617] 12:31:51,259 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,259 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower616: Last sync leader does not match current leader leader615, need to catch up to -1 [WARN] [08/01/2024 12:31:51.259] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/follower616] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4a5ecdc1 12:31:51,260 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,260 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,260 PM [test-pekko.actor.default-dispatcher-8] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618 (Follower): Term 2 in "AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 12:31:51,260 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Set currentTerm=2, votedFor=null 12:31:51,260 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 12:31:51,260 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] SyncStatusTracker - follower618: Last sync leader does not match current leader leader615, need to catch up to -1 12:31:51,260 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower617: Last sync leader does not match current leader leader615, need to catch up to -1 [WARN] [08/01/2024 12:31:51.260] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower617] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@23ca40d0 [WARN] [08/01/2024 12:31:51.260] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower618] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@71c5a342 12:31:51,260 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower616 updated: matchIndex: -1, nextIndex: 0 12:31:51,261 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,261 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower617 updated: matchIndex: -1, nextIndex: 0 12:31:51,261 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,261 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower618 updated: matchIndex: -1, nextIndex: 0 12:31:51,261 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,297 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 12:31:51,297 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 12:31:51,297 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 12:31:51,297 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower616] 12:31:51,298 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower618] 12:31:51,299 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 12:31:51,299 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 12:31:51,299 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: onRequestLeadership RequestLeadership [requestedFollowerId=follower617, replyTo=Actor[pekko://test/user/$d#1590215877]] 12:31:51,299 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Initiating leader transfer [WARN] [08/01/2024 12:31:51.299] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: LeaderTransitioning [leaderId=leader615] 12:31:51,299 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader615: pauseLeader successfully completed - doing transfer 12:31:51,299 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Attempting to transfer leadership [WARN] [08/01/2024 12:31:51.299] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: LeaderTransitioning [leaderId=leader615] 12:31:51,299 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: onLeaderTransitioning: LeaderTransitioning [leaderId=leader615] 12:31:51,299 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,299 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:51.299] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [08/01/2024 12:31:51.299] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:31:51,299 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,300 PM [test-pekko.actor.default-dispatcher-8] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 12:31:51,300 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): tryToCompleteLeadershipTransfer: followerId: follower617, matchIndex: -1, lastIndex: -1, isVoting: true 12:31:51,300 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Follower's log matches - sending ElectionTimeout 12:31:51,300 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:51.300] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower617] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@f414f65 12:31:51,300 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:51.300] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:31:51,300 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:51.300] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:31:51,300 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leader transfer complete 12:31:51,300 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader transfer complete - waiting for new leader 12:31:51,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 12:31:51,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 12:31:51,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Received TimeoutNow - switching to Candidate 12:31:51,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Election: Candidate has following voting peers: [follower616, leader615] 12:31:51,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=3, votedFor=follower617 12:31:51,300 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Starting new election term 3 12:31:51,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] to peer follower616 12:31:51,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] to peer leader615 12:31:51,300 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 [WARN] [08/01/2024 12:31:51.301] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/follower617]: RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] 12:31:51,301 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 12:31:51,301 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Cluster state: [] 12:31:51,301 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 12:31:51,301 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 12:31:51,301 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=3, votedFor=null 12:31:51,301 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leadership transfer in progress - processing RequestVote 12:31:51,301 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): In requestVote: RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 12:31:51,301 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=3, votedFor=follower617 12:31:51,301 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 12:31:51,301 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader) :- Switching from behavior Leader to Follower, election term: 3 12:31:51,301 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 12:31:51,301 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 12:31:51,301 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Election: Leader has following peers: [follower616, follower618, leader615] 12:31:51,301 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,301 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,301 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/01/2024 12:31:51.301] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/follower617]: AppendEntries [leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:31:51,301 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [08/01/2024 12:31:51.301] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/follower617]: AppendEntries [leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 12:31:51,302 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - leader615: Last sync leader does not match current leader follower617, need to catch up to -1 12:31:51,302 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader615 updated: matchIndex: -1, nextIndex: 0 12:31:51,302 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader changed to follower617 12:31:51,302 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 12:31:51,302 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorLeadershipTransferCohort - leader615: Successfully transferred leadership to follower617 in 3.480 ms 12:31:51,303 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Leadership transferred successfully to follower617 [WARN] [08/01/2024 12:31:51.303] [test-pekko.actor.default-dispatcher-8] [pekko://test/user/leader615] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6e26ea96 12:31:51,350 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 12:31:51,350 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower616-notifier619#1053966344] 12:31:51,351 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower616-collector620#-716227500] 12:31:51,351 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower617-notifier621#1953536129] 12:31:51,351 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower617-collector622#-531238812] 12:31:51,351 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower617] 12:31:51,351 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower618-notifier623#1662193087] 12:31:51,351 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower618-collector624#13828808] 12:31:51,351 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader615-notifier625#1380600005] 12:31:51,351 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader615-collector626#100307538] 12:31:51,352 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader615] 12:31:51,352 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$d#1590215877] [INFO] [08/01/2024 12:31:51.353] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 12:31:51,353 PM [test-pekko.actor.default-dispatcher-8] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 12:31:51,354 PM [test-pekko.actor.default-dispatcher-8] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.103:17355] - Exiting completed 12:31:51,354 PM [test-pekko.actor.default-dispatcher-8] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.103:17355] - Shutting down... 12:31:51,355 PM [test-pekko.actor.default-dispatcher-8] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.103:17355] - Successfully shut down 12:31:51,356 PM [test-pekko.actor.default-dispatcher-8] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 12:31:51,356 PM [test-pekko.actor.default-dispatcher-8] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [WARN] [08/01/2024 12:31:51.362] [test-pekko.actor.default-dispatcher-16] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.170.103:17355)