[DEBUG] [01/29/2025 19:25:51.658] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 07:25:51,659 PM [test-pekko.actor.default-dispatcher-4] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [01/29/2025 19:25:51.659] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [01/29/2025 19:25:51.659] [main] [EventStream(pekko://test)] Default Loggers started 07:25:51,669 PM [test-pekko.actor.default-dispatcher-4] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.247:17355] with UID [6784573098465472195] 07:25:51,670 PM [test-pekko.actor.default-dispatcher-4] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.247:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [01/29/2025 19:25:51.669] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.247:17355] with UID [6784573098465472195] [INFO] [01/29/2025 19:25:51.670] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.247:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [01/29/2025 19:25:51.673] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.247:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [01/29/2025 19:25:51.673] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.247:17355] - Started up successfully 07:25:51,673 PM [test-pekko.actor.default-dispatcher-4] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.247:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 07:25:51,673 PM [test-pekko.actor.default-dispatcher-4] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.247:17355] - Started up successfully 07:25:51,683 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.247:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [01/29/2025 19:25:51.683] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.247:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 07:25:51,683 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.247:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining [INFO] [01/29/2025 19:25:51.683] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.247:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 07:25:51,684 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 07:25:51,684 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/29/2025 19:25:51.691] [test-pekko.actor.default-dispatcher-4] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower551]: org.opendaylight.controller.cluster.common.actor.Monitor@184381ad [INFO] [pekkoDeadLetter][01/29/2025 19:25:51.690] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower551] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:51,691 PM [test-pekko.actor.default-dispatcher-4] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower551] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:51,693 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower551 with journal batch size 1000 07:25:51,693 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: SnapshotOffer called. 07:25:51,693 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Moving last applied index from -1 to -1 07:25:51,693 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower551: Set currentTerm=1, votedFor=null 07:25:51,693 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING]] [WARN] [01/29/2025 19:25:51.694] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower552]: org.opendaylight.controller.cluster.common.actor.Monitor@57a377d7 07:25:51,693 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower551 in 60.38 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [INFO] [pekkoDeadLetter][01/29/2025 19:25:51.694] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower552] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:51,694 PM [test-pekko.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower552] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:51,695 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower552 with journal batch size 1000 [WARN] [01/29/2025 19:25:51.696] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower553]: org.opendaylight.controller.cluster.common.actor.Monitor@179a6b90 [INFO] [pekkoDeadLetter][01/29/2025 19:25:51.696] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower553] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:51,697 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: SnapshotOffer called. 07:25:51,697 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Moving last applied index from -1 to -1 07:25:51,697 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - follower552: Set currentTerm=1, votedFor=null 07:25:51,697 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 07:25:51,697 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower552 in 66.51 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:51,697 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower553 with journal batch size 1000 07:25:51,697 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Recovery completed in 783.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:51,698 PM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower553] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:51,698 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: SnapshotOffer called. 07:25:51,698 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Moving last applied index from -1 to -1 07:25:51,698 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - follower553: Set currentTerm=1, votedFor=null 07:25:51,698 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Updated server config: isVoting: false, peers: [PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 07:25:51,698 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower553 in 48.86 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:51,699 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Recovery completed in 592.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:51.699] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader550]: org.opendaylight.controller.cluster.common.actor.Monitor@650acd94 [INFO] [pekkoDeadLetter][01/29/2025 19:25:51.699] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader550] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:51,700 PM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader550] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:51,700 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader550 with journal batch size 1000 07:25:51,701 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: SnapshotOffer called. 07:25:51,701 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Moving last applied index from -1 to -1 07:25:51,701 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - leader550: Set currentTerm=1, votedFor=null 07:25:51,701 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 07:25:51,701 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader550 in 56.47 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:51,702 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Recovery completed in 507.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:51,702 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Recovery completed in 9.198 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:52,80 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 07:25:52,80 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.247:17355, Removed) 07:25:52,81 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Received ElectionTimeout - switching to Candidate 07:25:52,81 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Election: Candidate has following voting peers: [follower552, follower551] 07:25:52,81 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - leader550: Set currentTerm=2, votedFor=leader550 07:25:52,81 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Starting new election term 2 07:25:52,81 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote{term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1} to peer follower552 07:25:52,81 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote{term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1} to peer follower551 07:25:52,81 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:52,81 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 07:25:52,81 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Cluster state: [] 07:25:52,81 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:52,81 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:52,81 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower552: Set currentTerm=2, votedFor=null 07:25:52,81 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:52,81 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Cluster state: [] 07:25:52,81 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): In requestVote: RequestVote{term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:52,81 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:52,81 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower552: Set currentTerm=2, votedFor=leader550 07:25:52,81 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:52,81 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - follower551: Set currentTerm=2, votedFor=null 07:25:52,81 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:52,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): In requestVote: RequestVote{term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:52,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - follower551: Set currentTerm=2, votedFor=leader550 07:25:52,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:52,83 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 07:25:52,83 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 07:25:52,84 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Election: Leader has following peers: [follower553, follower552, follower551] 07:25:52,84 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,84 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader550, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 07:25:52,84 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower553: Set currentTerm=2, votedFor=null 07:25:52,84 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower553: Last sync leader does not match current leader leader550, need to catch up to -1 07:25:52,84 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,84 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,84 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower551: Last sync leader does not match current leader leader550, need to catch up to -1 07:25:52,84 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower552: Last sync leader does not match current leader leader550, need to catch up to -1 [WARN] [01/29/2025 19:25:52.084] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower551] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5e085b0 [INFO] [pekkoDeadLetter][01/29/2025 19:25:52.085] [test-pekko.actor.default-dispatcher-4] [pekko://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:52,85 PM [test-pekko.actor.default-dispatcher-4] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [01/29/2025 19:25:52.085] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower552] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1def3152 07:25:52,85 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 07:25:52,85 PM [test-pekko.actor.default-dispatcher-11] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][01/29/2025 19:25:52.085] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:52,85 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower553 updated: matchIndex: -1, nextIndex: 0 07:25:52,85 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,85 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower551 updated: matchIndex: -1, nextIndex: 0 07:25:52,85 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,85 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower552 updated: matchIndex: -1, nextIndex: 0 07:25:52,85 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,86 PM [test-pekko.actor.default-dispatcher-16] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [01/29/2025 19:25:52.086] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower553] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@405f5cd5 [INFO] [pekkoDeadLetter][01/29/2025 19:25:52.086] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:52,106 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 07:25:52,106 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader550 starting 07:25:52,108 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: onShutDown 07:25:52,108 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Initiating leader transfer 07:25:52,109 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader550: pauseLeader successfully completed - doing transfer 07:25:52,109 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 07:25:52,109 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Attempting to transfer leadership 07:25:52,109 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,109 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 07:25:52,109 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 07:25:52,109 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [01/29/2025 19:25:52.109] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower553] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4eb657ac [INFO] [pekkoDeadLetter][01/29/2025 19:25:52.109] [test-pekko.actor.default-dispatcher-4] [pekko://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:52,109 PM [test-pekko.actor.default-dispatcher-4] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:52,109 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,109 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,109 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 07:25:52,109 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [01/29/2025 19:25:52.110] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower551] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@527c3f68 [WARN] [01/29/2025 19:25:52.110] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower552] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5330fd92 [INFO] [pekkoDeadLetter][01/29/2025 19:25:52.110] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:52,110 PM [test-pekko.actor.default-dispatcher-17] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][01/29/2025 19:25:52.110] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:52,110 PM [test-pekko.actor.default-dispatcher-17] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 07:25:52,201 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 07:25:52,201 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 07:25:52,201 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 07:25:52,321 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 07:25:52,321 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 07:25:52,321 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 07:25:52,440 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader550: leader transfer aborted 07:25:52,440 PM [test-pekko.actor.default-dispatcher-16] [WARN] RaftActorLeadershipTransferCohort - leader550: Failed to transfer leadership in 331.8 ms 07:25:52,440 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: leader transfer failed - sending PoisonPill 07:25:52,440 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Leadership transfer expired 07:25:52,441 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 07:25:52,441 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 07:25:52,441 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 07:25:52,443 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader550 ending 07:25:52,444 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 07:25:52,444 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower551-notifier554#1054817674] 07:25:52,456 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower551-collector555#-1194019514] 07:25:52,456 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower551] 07:25:52,456 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower552-notifier556#826242908] 07:25:52,456 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower552-collector557#1433902993] 07:25:52,456 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower552] 07:25:52,457 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower553-notifier558#-1594943574] 07:25:52,457 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower553-collector559#65830642] 07:25:52,457 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower553] 07:25:52,457 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader550-notifier560#-480938351] 07:25:52,458 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader550-collector561#977790568] 07:25:52,458 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader550] 07:25:52,459 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 07:25:52,459 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/29/2025 19:25:52.461] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower563]: org.opendaylight.controller.cluster.common.actor.Monitor@25f5e22d 07:25:52,462 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower563 with journal batch size 1000 07:25:52,462 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: SnapshotOffer called. 07:25:52,462 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Moving last applied index from -1 to -1 07:25:52,462 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower563: Set currentTerm=1, votedFor=null 07:25:52,462 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 07:25:52,462 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower563 in 100.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:52,463 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Recovery completed in 1.018 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:52.463] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower564]: org.opendaylight.controller.cluster.common.actor.Monitor@68bdc3d0 07:25:52,464 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower564 with journal batch size 1000 07:25:52,464 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: SnapshotOffer called. 07:25:52,464 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Moving last applied index from -1 to -1 07:25:52,464 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower564: Set currentTerm=1, votedFor=null 07:25:52,464 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 07:25:52,464 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower564 in 49.40 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:52,465 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Recovery completed in 490.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:52.465] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower565]: org.opendaylight.controller.cluster.common.actor.Monitor@27f980d3 07:25:52,466 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower565 with journal batch size 1000 07:25:52,466 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: SnapshotOffer called. 07:25:52,466 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Moving last applied index from -1 to -1 07:25:52,466 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - follower565: Set currentTerm=1, votedFor=null 07:25:52,466 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Updated server config: isVoting: false, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING]] 07:25:52,466 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower565 in 55.78 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:52,466 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Recovery completed in 468.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:52.568] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader562]: org.opendaylight.controller.cluster.common.actor.Monitor@7dc2f8ac 07:25:52,569 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader562 with journal batch size 1000 07:25:52,570 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: SnapshotOffer called. 07:25:52,570 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Moving last applied index from -1 to -1 07:25:52,570 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - leader562: Set currentTerm=1, votedFor=null 07:25:52,570 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Updated server config: isVoting: true, peers: [PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 07:25:52,570 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader562 in 83.00 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:52,571 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Recovery completed in 1.010 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:52,931 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 07:25:52,931 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.247:17355, Removed) 07:25:52,931 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Received ElectionTimeout - switching to Candidate 07:25:52,931 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Election: Candidate has following voting peers: [follower564, follower563] 07:25:52,931 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - leader562: Set currentTerm=2, votedFor=leader562 07:25:52,931 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Starting new election term 2 07:25:52,931 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote{term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1} to peer follower564 07:25:52,931 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote{term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1} to peer follower563 07:25:52,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:52,931 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 07:25:52,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Cluster state: [] 07:25:52,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:52,931 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:52,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - follower564: Set currentTerm=2, votedFor=null 07:25:52,931 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:52,932 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 07:25:52,932 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): In requestVote: RequestVote{term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:52,932 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:52,932 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - follower564: Set currentTerm=2, votedFor=leader562 07:25:52,932 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:52,932 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:52,932 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower563: Set currentTerm=2, votedFor=null 07:25:52,932 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 07:25:52,932 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 07:25:52,932 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote{term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:52,932 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower563: Set currentTerm=2, votedFor=leader562 07:25:52,933 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:52,933 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Election: Leader has following peers: [follower564, follower563, follower565] 07:25:52,933 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,933 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,933 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,933 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 07:25:52,933 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader562, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 07:25:52,933 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower565: Set currentTerm=2, votedFor=null 07:25:52,933 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader leader562, need to catch up to -1 07:25:52,933 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader leader562, need to catch up to -1 07:25:52,933 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 07:25:52,933 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:52.933] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6286e73c 07:25:52,934 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 07:25:52,934 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,935 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower564: Last sync leader does not match current leader leader562, need to catch up to -1 07:25:52,935 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower564 updated: matchIndex: -1, nextIndex: 0 07:25:52,935 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:52.935] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5c0d566 [WARN] [01/29/2025 19:25:52.935] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@601586bc 07:25:52,974 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 07:25:52,974 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 07:25:52,975 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 07:25:52,975 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 07:25:52,975 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 07:25:52,976 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: onRequestLeadership RequestLeadership [requestedFollowerId=follower564, replyTo=Actor[pekko://test/user/$a#2067897674]] 07:25:52,977 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Initiating leader transfer 07:25:52,978 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 07:25:52,978 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 07:25:52,978 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 07:25:52,978 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader562: pauseLeader successfully completed - doing transfer 07:25:52,978 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Attempting to transfer leadership 07:25:52,978 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,978 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,978 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,978 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 07:25:52,978 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 07:25:52,978 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) [WARN] [01/29/2025 19:25:52.978] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@71bcb52e [WARN] [01/29/2025 19:25:52.978] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@75833fc1 [WARN] [01/29/2025 19:25:52.978] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3081bf15 07:25:52,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): tryToCompleteLeadershipTransfer: followerId: follower564, matchIndex: -1, lastIndex: -1, isVoting: true 07:25:52,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Follower's log matches - sending ElectionTimeout 07:25:52,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,979 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 07:25:52,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leader transfer complete 07:25:52,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader transfer complete - waiting for new leader 07:25:52,979 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 07:25:52,979 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 07:25:52,979 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 07:25:52,979 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Received TimeoutNow - switching to Candidate 07:25:52,979 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Election: Candidate has following voting peers: [leader562, follower563] 07:25:52,979 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - follower564: Set currentTerm=3, votedFor=follower564 07:25:52,979 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Starting new election term 3 07:25:52,980 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote{term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1} to peer leader562 07:25:52,980 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote{term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1} to peer follower563 07:25:52,980 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:52,980 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Cluster state: [] 07:25:52,980 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 07:25:52,980 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:52,980 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:52,980 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower 07:25:52,980 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 07:25:52,980 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - leader562: Set currentTerm=3, votedFor=null 07:25:52,980 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:52,980 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 2 - updating term 07:25:52,980 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower563: Set currentTerm=3, votedFor=null 07:25:52,980 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leadership transfer in progress - processing RequestVote 07:25:52,980 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): In requestVote: RequestVote{term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:52,980 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - leader562: Set currentTerm=3, votedFor=follower564 07:25:52,980 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote{term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:52,980 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 07:25:52,980 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower563: Set currentTerm=3, votedFor=follower564 07:25:52,980 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 07:25:52,980 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader) :- Switching from behavior Leader to Follower, election term: 3 07:25:52,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 07:25:52,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 07:25:52,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Election: Leader has following peers: [leader562, follower563, follower565] 07:25:52,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,981 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 07:25:52,981 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader follower564, need to catch up to -1 07:25:52,981 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower564, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 07:25:52,981 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - follower565: Set currentTerm=3, votedFor=null 07:25:52,981 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader follower564, need to catch up to -1 [WARN] [01/29/2025 19:25:52.981] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@726346a3 07:25:52,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 07:25:52,982 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,982 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - leader562: Last sync leader does not match current leader follower564, need to catch up to -1 07:25:52,982 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader changed to follower564 07:25:52,982 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 07:25:52,982 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,982 PM [test-pekko.actor.default-dispatcher-4] [INFO] RaftActorLeadershipTransferCohort - leader562: Successfully transferred leadership to follower564 in 4.108 ms 07:25:52,982 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader562 updated: matchIndex: -1, nextIndex: 0 07:25:52,982 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:52,982 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Leadership transferred successfully to follower564 [WARN] [01/29/2025 19:25:52.982] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/leader562] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@157abb68 [WARN] [01/29/2025 19:25:52.983] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@49693e3b 07:25:53,27 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 07:25:53,27 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower563-notifier566#1787348681] 07:25:53,38 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower563-collector567#1440483188] 07:25:53,39 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower563] 07:25:53,39 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower564-notifier568#-1547602187] 07:25:53,39 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower564-collector569#-494994744] 07:25:53,39 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower564] 07:25:53,40 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower565-notifier570#-1941103132] 07:25:53,40 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower565-collector571#1049834844] 07:25:53,40 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower565] 07:25:53,40 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader562-notifier572#-1897719137] 07:25:53,40 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader562-collector573#-1266386817] 07:25:53,40 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader562] 07:25:53,41 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$a#2067897674] 07:25:53,42 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 07:25:53,42 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/29/2025 19:25:53.045] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower575]: org.opendaylight.controller.cluster.common.actor.Monitor@23e59849 07:25:53,45 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower575 with journal batch size 1000 07:25:53,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: SnapshotOffer called. 07:25:53,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Moving last applied index from -1 to -1 07:25:53,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - follower575: Set currentTerm=1, votedFor=null 07:25:53,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 07:25:53,47 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower575 in 113.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:53,48 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Recovery completed in 1.102 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:53.146] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower576]: org.opendaylight.controller.cluster.common.actor.Monitor@10b54edf 07:25:53,147 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower576 with journal batch size 1000 07:25:53,147 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: SnapshotOffer called. 07:25:53,148 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Moving last applied index from -1 to -1 07:25:53,148 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - follower576: Set currentTerm=1, votedFor=null 07:25:53,148 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING]] 07:25:53,148 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower576 in 95.93 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [01/29/2025 19:25:53.149] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower577]: org.opendaylight.controller.cluster.common.actor.Monitor@573f9856 07:25:53,149 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Recovery completed in 886.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:53,149 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower577 with journal batch size 1000 07:25:53,150 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: SnapshotOffer called. 07:25:53,150 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Moving last applied index from -1 to -1 07:25:53,150 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower577: Set currentTerm=1, votedFor=null 07:25:53,150 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Updated server config: isVoting: false, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 07:25:53,150 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower577 in 78.79 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:53,151 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Recovery completed in 790.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:53.151] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader574]: org.opendaylight.controller.cluster.common.actor.Monitor@6f6d38e2 07:25:53,152 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader574 with journal batch size 1000 07:25:53,152 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: SnapshotOffer called. 07:25:53,152 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Moving last applied index from -1 to -1 07:25:53,152 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - leader574: Set currentTerm=1, votedFor=null 07:25:53,152 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Updated server config: isVoting: true, peers: [PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 07:25:53,152 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader574 in 45.55 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:53,152 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Recovery completed in 523.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:53,501 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 07:25:53,501 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.247:17355, Removed) 07:25:53,501 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Received ElectionTimeout - switching to Candidate 07:25:53,501 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Election: Candidate has following voting peers: [follower575, follower576] 07:25:53,501 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - leader574: Set currentTerm=2, votedFor=leader574 07:25:53,501 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Starting new election term 2 07:25:53,501 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote{term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1} to peer follower575 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote{term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1} to peer follower576 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Cluster state: [] 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower575: Set currentTerm=2, votedFor=null 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): In requestVote: RequestVote{term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower575: Set currentTerm=2, votedFor=leader574 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Cluster state: [] 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower576: Set currentTerm=2, votedFor=null 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): In requestVote: RequestVote{term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:53,502 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower576: Set currentTerm=2, votedFor=leader574 07:25:53,503 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:53,503 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 07:25:53,503 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 07:25:53,503 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Election: Leader has following peers: [follower575, follower577, follower576] 07:25:53,503 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:53,503 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:53,503 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:53,503 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower575: Last sync leader does not match current leader leader574, need to catch up to -1 07:25:53,504 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 [WARN] [01/29/2025 19:25:53.504] [test-pekko.actor.default-dispatcher-4] [pekko://test/user/follower575] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@600a345b 07:25:53,504 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 07:25:53,504 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - follower577: Set currentTerm=2, votedFor=null 07:25:53,504 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower575 updated: matchIndex: -1, nextIndex: 0 07:25:53,504 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower577: Last sync leader does not match current leader leader574, need to catch up to -1 07:25:53,504 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:53,504 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower577 updated: matchIndex: -1, nextIndex: 0 07:25:53,504 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:53,505 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower576: Last sync leader does not match current leader leader574, need to catch up to -1 07:25:53,505 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower576 updated: matchIndex: -1, nextIndex: 0 07:25:53,505 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:53.505] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower576] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4bfb19cf [WARN] [01/29/2025 19:25:53.505] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower577] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1a8289a4 07:25:53,505 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 07:25:53,505 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 07:25:53,506 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 07:25:53,506 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower576 starting 07:25:53,506 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: onShutDown 07:25:53,506 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower576 ending 07:25:53,506 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 07:25:53,506 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 07:25:53,506 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: onRequestLeadership RequestLeadership [requestedFollowerId=follower576, replyTo=Actor[pekko://test/user/$b#-1398070015]] 07:25:53,506 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Initiating leader transfer 07:25:53,507 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] 07:25:53,507 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader574: pauseLeader successfully completed - doing transfer 07:25:53,507 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Attempting to transfer leadership 07:25:53,507 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:53,507 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:53,507 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:53.507] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: LeaderTransitioning [leaderId=leader574] 07:25:53,507 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [01/29/2025 19:25:53.507] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries{term=2, leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [01/29/2025 19:25:53.507] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower575] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3d679e87 07:25:53,507 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 07:25:53,507 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] [WARN] [01/29/2025 19:25:53.508] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower577] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@555571f0 07:25:53,621 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [01/29/2025 19:25:53.621] [test-pekko.actor.default-dispatcher-4] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries{term=2, leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 07:25:53,621 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 07:25:53,741 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [01/29/2025 19:25:53.741] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries{term=2, leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 07:25:53,741 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 07:25:53,860 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader574: leader transfer aborted 07:25:53,860 PM [test-pekko.actor.default-dispatcher-13] [WARN] RaftActorLeadershipTransferCohort - leader574: Failed to transfer leadership in 353.5 ms 07:25:53,860 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: LeadershipTransfer request from follower576 failed 07:25:53,861 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Leadership transfer expired 07:25:53,861 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 07:25:53,861 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [01/29/2025 19:25:53.861] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries{term=2, leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 07:25:53,910 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 07:25:53,910 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower575-notifier578#2091941062] 07:25:53,911 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower575-collector579#-182566595] 07:25:53,912 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower575] 07:25:53,912 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower576-notifier580#-535878708] 07:25:53,912 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower576-collector581#455293625] 07:25:53,912 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower576] 07:25:53,912 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower577-notifier582#-1449535376] 07:25:53,913 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower577-collector583#-553539688] 07:25:53,913 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower577] 07:25:53,913 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader574-notifier584#-697991250] 07:25:53,913 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader574-collector585#-1975402109] 07:25:53,913 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader574] 07:25:53,914 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$b#-1398070015] 07:25:53,915 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 07:25:53,915 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/29/2025 19:25:53.917] [test-pekko.actor.default-dispatcher-4] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower587]: org.opendaylight.controller.cluster.common.actor.Monitor@389cd3ff 07:25:53,918 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower587 with journal batch size 1000 07:25:53,919 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: SnapshotOffer called. 07:25:53,919 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to -1 07:25:53,919 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - follower587: Set currentTerm=1, votedFor=null 07:25:53,919 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING]] [WARN] [01/29/2025 19:25:53.920] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower588]: org.opendaylight.controller.cluster.common.actor.Monitor@7a018546 07:25:53,920 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower587 in 97.56 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:53,920 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower588 with journal batch size 1000 07:25:53,920 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Recovery completed in 811.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:53,921 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: SnapshotOffer called. 07:25:53,921 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to -1 07:25:53,921 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - follower588: Set currentTerm=1, votedFor=null 07:25:53,921 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 07:25:53,921 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower588 in 64.18 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:53,921 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Recovery completed in 652.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:53.922] [test-pekko.actor.default-dispatcher-4] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower589]: org.opendaylight.controller.cluster.common.actor.Monitor@f16a988 07:25:53,923 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower589 with journal batch size 1000 07:25:53,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: SnapshotOffer called. 07:25:53,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to -1 07:25:53,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - follower589: Set currentTerm=1, votedFor=null 07:25:53,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Updated server config: isVoting: false, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 07:25:53,923 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower589 in 39.78 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:53,924 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Recovery completed in 587.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:53.924] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader586]: org.opendaylight.controller.cluster.common.actor.Monitor@60e7bf27 07:25:53,925 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader586 with journal batch size 1000 07:25:53,925 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: SnapshotOffer called. 07:25:53,925 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to -1 07:25:53,925 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - leader586: Set currentTerm=1, votedFor=null 07:25:53,925 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Updated server config: isVoting: true, peers: [PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 07:25:53,925 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader586 in 44.58 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:53,925 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Recovery completed in 426.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:54,320 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 07:25:54,320 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.247:17355, Removed) 07:25:54,320 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Received ElectionTimeout - switching to Candidate 07:25:54,320 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Election: Candidate has following voting peers: [follower588, follower587] 07:25:54,320 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - leader586: Set currentTerm=2, votedFor=leader586 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Starting new election term 2 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote{term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1} to peer follower588 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote{term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1} to peer follower587 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - follower588: Set currentTerm=2, votedFor=null 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote{term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:54,321 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - follower588: Set currentTerm=2, votedFor=leader586 07:25:54,321 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:54,322 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Cluster state: [] 07:25:54,322 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:54,322 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:54,322 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - follower587: Set currentTerm=2, votedFor=null 07:25:54,322 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): In requestVote: RequestVote{term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:54,322 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - follower587: Set currentTerm=2, votedFor=leader586 07:25:54,322 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:54,322 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 07:25:54,322 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 07:25:54,322 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Election: Leader has following peers: [follower589, follower588, follower587] 07:25:54,322 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:54,322 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:54,323 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:54,323 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:54,323 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 07:25:54,323 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - follower589: Set currentTerm=2, votedFor=null 07:25:54,323 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 07:25:54,323 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader leader586, need to catch up to -1 07:25:54,323 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: -1, nextIndex: 0 07:25:54,323 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:54,323 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader leader586, need to catch up to -1 07:25:54,323 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: -1, nextIndex: 0 07:25:54,323 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:54.323] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@691bb789 [WARN] [01/29/2025 19:25:54.324] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6c6586f 07:25:54,324 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower587: Last sync leader does not match current leader leader586, need to catch up to -1 [WARN] [01/29/2025 19:25:54.324] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@41da1d8d 07:25:54,324 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: -1, nextIndex: 0 07:25:54,324 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:54,328 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 07:25:54,328 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 07:25:54,328 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:54,328 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:54,328 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 07:25:54,329 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,329 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower589 07:25:54,329 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower589: AppendEntries{term=2, leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Number of entries to be appended = 1 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Append entry to log zero 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Log size is now 1 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower589, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 07:25:54,329 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,329 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 07:25:54,329 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries{term=2, leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:54,329 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,329 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower587 07:25:54,329 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower587: AppendEntries{term=2, leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Number of entries to be appended = 1 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Append entry to log zero 07:25:54,329 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [01/29/2025 19:25:54.330] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@38ec5a60 07:25:54,329 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:54,330 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Log size is now 1 07:25:54,330 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower587, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 07:25:54,330 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 07:25:54,330 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 07:25:54,330 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,330 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: 0, nextIndex: 1 07:25:54,330 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,330 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:54,331 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [01/29/2025 19:25:54.331] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@22ee21c 07:25:54,331 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 07:25:54,331 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Setting last applied to 0 07:25:54,331 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to 0 07:25:54,331 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Applying state for log index 0 data zero 07:25:54,331 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: applyState called: zero 07:25:54,331 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persisting ApplyJournalEntries with index=0 07:25:54,441 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 07:25:54,441 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries{term=2, leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:54,441 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Commit index set to 0 07:25:54,441 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Commit index set to 0 07:25:54,441 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 07:25:54,441 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 07:25:54,441 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Setting last applied to 0 07:25:54,441 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to 0 07:25:54,441 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Applying state for log index 0 data zero 07:25:54,441 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: applyState called: zero 07:25:54,441 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Persisting ApplyJournalEntries with index=0 07:25:54,441 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 07:25:54,441 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 07:25:54,441 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Setting last applied to 0 07:25:54,441 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to 0 07:25:54,441 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Applying state for log index 0 data zero 07:25:54,442 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: applyState called: zero 07:25:54,442 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Persisting ApplyJournalEntries with index=0 07:25:54,479 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 07:25:54,479 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 07:25:54,479 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: onShutDown 07:25:54,479 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Initiating leader transfer 07:25:54,480 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 07:25:54,480 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader586: pauseLeader successfully completed - doing transfer 07:25:54,480 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 07:25:54,480 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Attempting to transfer leadership 07:25:54,480 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,480 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 07:25:54,480 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,480 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 07:25:54,480 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,480 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower589, matchIndex: 0, lastIndex: 0, isVoting: false 07:25:54,480 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 07:25:54,480 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower587, matchIndex: 0, lastIndex: 0, isVoting: true 07:25:54,480 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Follower's log matches - sending ElectionTimeout 07:25:54,481 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,481 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 07:25:54,481 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,481 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 07:25:54,481 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,481 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leader transfer complete 07:25:54,481 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader transfer complete - waiting for new leader 07:25:54,481 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 07:25:54,481 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 07:25:54,481 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Received TimeoutNow - switching to Candidate 07:25:54,481 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Election: Candidate has following voting peers: [leader586, follower588] 07:25:54,481 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower587: Set currentTerm=3, votedFor=follower587 07:25:54,481 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Starting new election term 3 07:25:54,481 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote{term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2} to peer leader586 07:25:54,481 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote{term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2} to peer follower588 07:25:54,481 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 07:25:54,482 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:54,482 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 07:25:54,482 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 07:25:54,482 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2}" message is greater than follower's term 2 - updating term 07:25:54,482 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower588: Set currentTerm=3, votedFor=null 07:25:54,482 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote{term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:54,482 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower588: Set currentTerm=3, votedFor=follower587 07:25:54,482 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 07:25:54,482 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:54,482 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Cluster state: [] 07:25:54,482 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 07:25:54,482 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2}" message is greater than leader's term 2 - switching to Follower 07:25:54,482 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - leader586: Set currentTerm=3, votedFor=null 07:25:54,482 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leadership transfer in progress - processing RequestVote 07:25:54,482 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): In requestVote: RequestVote{term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 07:25:54,482 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - leader586: Set currentTerm=3, votedFor=follower587 07:25:54,482 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 07:25:54,482 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 07:25:54,482 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 07:25:54,482 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Election: Leader has following peers: [leader586, follower589, follower588] 07:25:54,482 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,482 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader) :- Switching from behavior Leader to Follower, election term: 3 07:25:54,482 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,483 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,483 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 07:25:54,483 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 07:25:54,483 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower589: Set currentTerm=3, votedFor=null 07:25:54,483 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader follower587, need to catch up to -1 07:25:54,483 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 07:25:54,483 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:54,485 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader586 updated: matchIndex: 0, nextIndex: 1 07:25:54,485 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 07:25:54,485 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:54.485] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7875e3b5 [WARN] [01/29/2025 19:25:54.485] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@17725e86 07:25:54,485 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - leader586: Last sync leader does not match current leader follower587, need to catch up to -1 07:25:54,486 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader changed to follower587 07:25:54,486 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorLeadershipTransferCohort - leader586: Successfully transferred leadership to follower587 in 6.016 ms 07:25:54,486 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: leader transfer succeeded - sending PoisonPill 07:25:54,486 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - leader586: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) [WARN] [01/29/2025 19:25:54.486] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader586] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@595ca8d [WARN] [01/29/2025 19:25:54.486] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/leader586] received dead letter: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1eac315 07:25:55,1 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) [WARN] [01/29/2025 19:25:55.001] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader586] received dead letter from TestActor[pekko://test/user/follower587]: AppendEntries{term=3, leaderId=follower587, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 07:25:55,1 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader follower587, need to catch up to -1 07:25:55,1 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): sendAppendEntries: 0 is present for follower follower588 07:25:55,2 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Sending AppendEntries to follower follower588: AppendEntries{term=3, leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} [WARN] [01/29/2025 19:25:55.002] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@35798b5e 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Number of entries to be appended = 1 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Append entry to log zero 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Log size is now 1 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Commit index set to 0 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower588, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Setting last applied to 0 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to 0 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Applying state for log index 0 data zero 07:25:55,2 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: 0, nextIndex: 1 07:25:55,2 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: applyState called: zero 07:25:55,2 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:55,3 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:55,3 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Persisting ApplyJournalEntries with index=0 [WARN] [01/29/2025 19:25:55.003] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2ff8fcd8 07:25:55,33 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 07:25:55,34 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 07:25:55,34 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower588 starting 07:25:55,34 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onShutDown 07:25:55,35 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower588 ending 07:25:55,35 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 07:25:55,35 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587-notifier590#501327499] 07:25:55,36 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587-collector591#-1429667985] 07:25:55,36 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower587] 07:25:55,36 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower588-notifier592#168462141] 07:25:55,37 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower588-collector593#-1252811747] 07:25:55,37 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower588] 07:25:55,37 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower589-notifier594#-746470862] 07:25:55,37 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower589-collector595#-468357587] 07:25:55,37 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower589] 07:25:55,38 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader586-notifier596#1081251308] 07:25:55,38 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader586-collector597#310628024] 07:25:55,38 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader586] 07:25:55,40 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [01/29/2025 19:25:55.041] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader598]: org.opendaylight.controller.cluster.common.actor.Monitor@4dce5d71 07:25:55,42 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader598 with journal batch size 1000 07:25:55,43 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:55,44 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 07:25:55,44 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower): Received TimeoutNow - switching to Candidate 07:25:55,44 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Election: Candidate has following voting peers: [] 07:25:55,44 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - leader598: Set currentTerm=1, votedFor=leader598 07:25:55,44 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Starting new election term 1 07:25:55,44 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 07:25:55,45 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Received ElectionTimeout 07:25:55,45 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Leader): Election: Leader has following peers: [] 07:25:55,45 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 07:25:55,45 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader598 starting 07:25:55,45 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: onShutDown 07:25:55,46 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader598 ending 07:25:55,46 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 07:25:55,46 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader598-collector602#1929816583] 07:25:55,47 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader598] 07:25:55,47 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 07:25:55,47 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/29/2025 19:25:55.049] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower604]: org.opendaylight.controller.cluster.common.actor.Monitor@edbab3 07:25:55,50 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower604 with journal batch size 1000 07:25:55,50 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: SnapshotOffer called. 07:25:55,50 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to -1 07:25:55,50 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower604: Set currentTerm=1, votedFor=null 07:25:55,50 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 07:25:55,50 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower604 in 149.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [01/29/2025 19:25:55.051] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower605]: org.opendaylight.controller.cluster.common.actor.Monitor@268a92ae 07:25:55,51 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Recovery completed in 973.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:55,52 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower605 with journal batch size 1000 [WARN] [01/29/2025 19:25:55.053] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower606]: org.opendaylight.controller.cluster.common.actor.Monitor@43a5d447 07:25:55,53 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: SnapshotOffer called. 07:25:55,53 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Moving last applied index from -1 to -1 07:25:55,53 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - follower605: Set currentTerm=1, votedFor=null 07:25:55,53 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower606 with journal batch size 1000 07:25:55,53 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Updated server config: isVoting: true, peers: [PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 07:25:55,53 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower605 in 68.35 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:55,53 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: SnapshotOffer called. 07:25:55,53 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to -1 07:25:55,53 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower606: Set currentTerm=1, votedFor=null 07:25:55,53 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Recovery completed in 436.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:55,53 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Updated server config: isVoting: false, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 07:25:55,53 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower606 in 60.86 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:55,54 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Recovery completed in 598.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:55.055] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader603]: org.opendaylight.controller.cluster.common.actor.Monitor@30df0ae6 07:25:55,55 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader603 with journal batch size 1000 07:25:55,55 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: SnapshotOffer called. 07:25:55,56 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to -1 07:25:55,56 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - leader603: Set currentTerm=1, votedFor=null 07:25:55,56 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING]] 07:25:55,56 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader603 in 30.42 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:55,56 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Recovery completed in 331.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:55,470 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 07:25:55,470 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.247:17355, Removed) 07:25:55,470 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Received ElectionTimeout - switching to Candidate 07:25:55,471 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Election: Candidate has following voting peers: [follower605, follower604] 07:25:55,471 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - leader603: Set currentTerm=2, votedFor=leader603 07:25:55,471 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Starting new election term 2 07:25:55,471 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote{term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1} to peer follower605 07:25:55,471 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote{term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1} to peer follower604 07:25:55,471 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:55,471 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 07:25:55,471 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Cluster state: [] 07:25:55,471 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:55,471 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:55,471 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower605: Set currentTerm=2, votedFor=null 07:25:55,471 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:55,471 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): In requestVote: RequestVote{term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:55,471 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower605: Set currentTerm=2, votedFor=leader603 07:25:55,471 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Cluster state: [] 07:25:55,471 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:55,471 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:55,471 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:55,471 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - follower604: Set currentTerm=2, votedFor=null 07:25:55,471 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): In requestVote: RequestVote{term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:55,471 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - follower604: Set currentTerm=2, votedFor=leader603 07:25:55,472 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:55,472 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 07:25:55,472 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 07:25:55,472 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Election: Leader has following peers: [follower605, follower604, follower606] 07:25:55,472 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:55,472 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:55,472 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:55,473 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 07:25:55,473 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 07:25:55,473 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower606: Set currentTerm=2, votedFor=null 07:25:55,473 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower605: Last sync leader does not match current leader leader603, need to catch up to -1 07:25:55,473 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower606: Last sync leader does not match current leader leader603, need to catch up to -1 07:25:55,473 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower605 updated: matchIndex: -1, nextIndex: 0 07:25:55,473 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower604: Last sync leader does not match current leader leader603, need to catch up to -1 07:25:55,473 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:55,473 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: -1, nextIndex: 0 [WARN] [01/29/2025 19:25:55.473] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower605] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@c8af223 [WARN] [01/29/2025 19:25:55.473] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower604] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3cf914c7 07:25:55,473 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:55.473] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower606] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6c2adb39 07:25:55,473 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: -1, nextIndex: 0 07:25:55,474 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:55,510 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 07:25:55,510 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 07:25:55,510 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 07:25:55,510 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries{term=2, leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower604 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower604: AppendEntries{term=2, leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower606 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower606: AppendEntries{term=2, leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Number of entries to be appended = 1 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Append entry to log zero 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Log size is now 1 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower604, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 07:25:55,511 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 07:25:55,512 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:55,512 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Number of entries to be appended = 1 07:25:55,512 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 07:25:55,512 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: 0, nextIndex: 1 07:25:55,512 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:55,512 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:55,512 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 07:25:55,512 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Setting last applied to 0 07:25:55,512 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to 0 07:25:55,512 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Applying state for log index 0 data zero 07:25:55,512 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: applyState called: zero 07:25:55,512 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 07:25:55,512 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persisting ApplyJournalEntries with index=0 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Append entry to log zero 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [01/29/2025 19:25:55.513] [test-pekko.actor.default-dispatcher-4] [pekko://test/user/follower604] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6de5325 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Log size is now 1 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower606, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: 0, nextIndex: 1 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [01/29/2025 19:25:55.513] [test-pekko.actor.default-dispatcher-4] [pekko://test/user/follower606] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@26fe1134 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Commit index set to 0 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Setting last applied to 0 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to 0 07:25:55,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Applying state for log index 0 data zero 07:25:55,514 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: applyState called: zero 07:25:55,514 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Persisting ApplyJournalEntries with index=0 07:25:55,590 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 07:25:55,591 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries{term=2, leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:55,710 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 07:25:55,710 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries{term=2, leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 07:25:55,711 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Commit index set to 0 07:25:55,711 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:55,711 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 07:25:55,711 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Setting last applied to 0 07:25:55,711 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to 0 07:25:55,711 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Applying state for log index 0 data zero 07:25:55,711 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: applyState called: zero 07:25:55,711 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Persisting ApplyJournalEntries with index=0 07:25:55,711 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:55,715 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 07:25:55,715 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 07:25:55,715 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 07:25:55,715 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: onRequestLeadership RequestLeadership [requestedFollowerId=follower605, replyTo=Actor[pekko://test/user/$c#7164381]] 07:25:55,715 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Initiating leader transfer 07:25:55,716 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 07:25:55,716 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 07:25:55,716 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 07:25:55,716 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader603: pauseLeader successfully completed - doing transfer 07:25:55,716 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Attempting to transfer leadership 07:25:55,716 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:55,716 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 07:25:55,716 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:55,716 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 07:25:55,716 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:55,716 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:55,831 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:55,831 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:55,951 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:55,951 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:56,70 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader603: leader transfer aborted 07:25:56,70 PM [test-pekko.actor.default-dispatcher-11] [WARN] RaftActorLeadershipTransferCohort - leader603: Failed to transfer leadership in 354.7 ms 07:25:56,70 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: LeadershipTransfer request from follower605 failed 07:25:56,70 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Leadership transfer expired 07:25:56,71 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:56,71 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 07:25:56,119 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 07:25:56,119 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower604-notifier607#-156747328] 07:25:56,121 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower604-collector608#-1016269181] 07:25:56,121 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower604] 07:25:56,121 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower605-notifier609#1200341937] 07:25:56,121 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower605-collector610#-363718025] 07:25:56,122 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower605] 07:25:56,122 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower606-notifier611#-2120647614] 07:25:56,122 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower606-collector612#238602012] 07:25:56,122 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower606] 07:25:56,122 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader603-notifier613#-1253269023] 07:25:56,122 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader603-collector614#-1661947885] 07:25:56,123 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader603] 07:25:56,123 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$c#7164381] 07:25:56,124 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 07:25:56,124 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/29/2025 19:25:56.126] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower616]: org.opendaylight.controller.cluster.common.actor.Monitor@716e9938 07:25:56,127 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower616 with journal batch size 1000 07:25:56,128 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: SnapshotOffer called. 07:25:56,128 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Moving last applied index from -1 to -1 07:25:56,128 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - follower616: Set currentTerm=1, votedFor=null 07:25:56,128 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Updated server config: isVoting: true, peers: [PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 07:25:56,128 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower616 in 102.8 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:56,129 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Recovery completed in 915.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:56.129] [test-pekko.actor.default-dispatcher-4] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower617]: org.opendaylight.controller.cluster.common.actor.Monitor@4e0283d8 07:25:56,129 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower617 with journal batch size 1000 07:25:56,130 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: SnapshotOffer called. 07:25:56,130 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Moving last applied index from -1 to -1 07:25:56,130 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - follower617: Set currentTerm=1, votedFor=null 07:25:56,130 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 07:25:56,130 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower617 in 121.8 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:56,130 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Recovery completed in 544.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:56.131] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower618]: org.opendaylight.controller.cluster.common.actor.Monitor@6452dd85 07:25:56,132 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower618 with journal batch size 1000 07:25:56,132 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: SnapshotOffer called. 07:25:56,132 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Moving last applied index from -1 to -1 07:25:56,132 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - follower618: Set currentTerm=1, votedFor=null 07:25:56,132 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Updated server config: isVoting: false, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 07:25:56,132 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower618 in 73.93 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:56,133 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Recovery completed in 495.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [01/29/2025 19:25:56.133] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader615]: org.opendaylight.controller.cluster.common.actor.Monitor@7d779334 07:25:56,133 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader615 with journal batch size 1000 07:25:56,134 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: SnapshotOffer called. 07:25:56,134 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Moving last applied index from -1 to -1 07:25:56,134 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - leader615: Set currentTerm=1, votedFor=null 07:25:56,134 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING]] 07:25:56,134 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader615 in 72.68 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 07:25:56,134 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Recovery completed in 498.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 07:25:56,471 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 07:25:56,471 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.247:17355, Removed) 07:25:56,471 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Received ElectionTimeout - switching to Candidate 07:25:56,471 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Election: Candidate has following voting peers: [follower616, follower617] 07:25:56,471 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - leader615: Set currentTerm=2, votedFor=leader615 07:25:56,471 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Starting new election term 2 07:25:56,471 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote{term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1} to peer follower616 07:25:56,471 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote{term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1} to peer follower617 07:25:56,471 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 07:25:56,471 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:56,472 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:56,472 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Cluster state: [] 07:25:56,472 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:56,472 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:56,472 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower616: Set currentTerm=2, votedFor=null 07:25:56,472 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Cluster state: [] 07:25:56,472 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:56,472 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 07:25:56,472 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - follower617: Set currentTerm=2, votedFor=null 07:25:56,472 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): In requestVote: RequestVote{term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:56,472 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower616: Set currentTerm=2, votedFor=leader615 07:25:56,472 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): In requestVote: RequestVote{term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:56,472 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:56,472 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PersistenceTermInfoStore - follower617: Set currentTerm=2, votedFor=leader615 07:25:56,472 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 07:25:56,472 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 07:25:56,472 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 07:25:56,473 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Election: Leader has following peers: [follower616, follower618, follower617] 07:25:56,473 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,473 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,473 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,473 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower616: Last sync leader does not match current leader leader615, need to catch up to -1 07:25:56,473 PM [test-pekko.actor.default-dispatcher-4] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 07:25:56,473 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 07:25:56,473 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] PersistenceTermInfoStore - follower618: Set currentTerm=2, votedFor=null 07:25:56,473 PM [test-pekko.actor.default-dispatcher-4] [DEBUG] SyncStatusTracker - follower618: Last sync leader does not match current leader leader615, need to catch up to -1 07:25:56,473 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower616 updated: matchIndex: -1, nextIndex: 0 07:25:56,473 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,473 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower618 updated: matchIndex: -1, nextIndex: 0 07:25:56,474 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:56.473] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower616] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1beb54fd [WARN] [01/29/2025 19:25:56.474] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower618] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2740b34c 07:25:56,475 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower617: Last sync leader does not match current leader leader615, need to catch up to -1 07:25:56,475 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower617 updated: matchIndex: -1, nextIndex: 0 07:25:56,475 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:56.475] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower617] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@fe60442 07:25:56,500 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 07:25:56,500 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 07:25:56,501 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 07:25:56,501 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower616] 07:25:56,502 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower618] 07:25:56,502 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 07:25:56,502 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 07:25:56,502 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: onRequestLeadership RequestLeadership [requestedFollowerId=follower617, replyTo=Actor[pekko://test/user/$d#649532732]] 07:25:56,502 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Initiating leader transfer 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader615: pauseLeader successfully completed - doing transfer 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Attempting to transfer leadership 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:56.503] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: LeaderTransitioning [leaderId=leader615] [WARN] [01/29/2025 19:25:56.503] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: LeaderTransitioning [leaderId=leader615] [WARN] [01/29/2025 19:25:56.503] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries{term=2, leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [01/29/2025 19:25:56.503] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries{term=2, leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: onLeaderTransitioning: LeaderTransitioning [leaderId=leader615] 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): tryToCompleteLeadershipTransfer: followerId: follower617, matchIndex: -1, lastIndex: -1, isVoting: true 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Follower's log matches - sending ElectionTimeout 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leader transfer complete 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader transfer complete - waiting for new leader [WARN] [01/29/2025 19:25:56.503] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower617] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2602fb47 [WARN] [01/29/2025 19:25:56.503] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries{term=2, leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [01/29/2025 19:25:56.503] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries{term=2, leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Received TimeoutNow - switching to Candidate 07:25:56,503 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Election: Candidate has following voting peers: [follower616, leader615] 07:25:56,504 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower617: Set currentTerm=3, votedFor=follower617 07:25:56,504 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Starting new election term 3 07:25:56,504 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote{term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1} to peer follower616 07:25:56,504 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote{term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1} to peer leader615 07:25:56,504 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 [WARN] [01/29/2025 19:25:56.504] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/follower617]: RequestVote{term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1} 07:25:56,504 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 07:25:56,504 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Cluster state: [] 07:25:56,504 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 07:25:56,504 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower 07:25:56,504 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - leader615: Set currentTerm=3, votedFor=null 07:25:56,504 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leadership transfer in progress - processing RequestVote 07:25:56,504 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): In requestVote: RequestVote{term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 07:25:56,504 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - leader615: Set currentTerm=3, votedFor=follower617 07:25:56,504 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 07:25:56,504 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader) :- Switching from behavior Leader to Follower, election term: 3 07:25:56,504 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 07:25:56,504 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 07:25:56,504 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Election: Leader has following peers: [follower616, follower618, leader615] 07:25:56,504 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,504 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,505 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 07:25:56,505 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [01/29/2025 19:25:56.505] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/follower617]: AppendEntries{term=3, leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 07:25:56,505 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - leader615: Last sync leader does not match current leader follower617, need to catch up to -1 07:25:56,505 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader changed to follower617 07:25:56,505 PM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorLeadershipTransferCohort - leader615: Successfully transferred leadership to follower617 in 2.313 ms 07:25:56,505 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Leadership transferred successfully to follower617 07:25:56,505 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader615 updated: matchIndex: -1, nextIndex: 0 07:25:56,505 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/29/2025 19:25:56.505] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/follower617]: AppendEntries{term=3, leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} [WARN] [01/29/2025 19:25:56.505] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/leader615] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4444a6d 07:25:56,555 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 07:25:56,555 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower616-notifier619#1400830270] 07:25:56,566 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower616-collector620#842786691] 07:25:56,566 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower617-notifier621#-373701774] 07:25:56,567 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower617-collector622#-1275543959] 07:25:56,567 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower617] 07:25:56,567 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower618-notifier623#914212401] 07:25:56,568 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower618-collector624#-1053642885] 07:25:56,568 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader615-notifier625#1053313322] 07:25:56,568 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader615-collector626#-150523877] 07:25:56,568 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader615] 07:25:56,568 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$d#649532732] 07:25:56,570 PM [test-pekko.actor.default-dispatcher-13] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 07:25:56,572 PM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.247:17355] - Exiting completed 07:25:56,572 PM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.247:17355] - Shutting down... [INFO] [01/29/2025 19:25:56.570] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [01/29/2025 19:25:56.572] [test-pekko.actor.default-dispatcher-18] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.170.247:17355) [INFO] [01/29/2025 19:25:56.571] [test-pekko.actor.internal-dispatcher-8] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.247:17355] - Exiting completed 07:25:56,572 PM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.247:17355] - Successfully shut down 07:25:56,572 PM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 07:25:56,572 PM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [01/29/2025 19:25:56.571] [test-pekko.actor.internal-dispatcher-8] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.247:17355] - Shutting down... [INFO] [01/29/2025 19:25:56.571] [test-pekko.actor.internal-dispatcher-8] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.247:17355] - Successfully shut down [INFO] [01/29/2025 19:25:56.572] [test-pekko.remote.default-remote-dispatcher-7] [pekko://test@10.30.170.247:17355/system/remoting-terminator] Shutting down remote daemon. [INFO] [01/29/2025 19:25:56.572] [test-pekko.remote.default-remote-dispatcher-7] [pekko://test@10.30.170.247:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. [INFO] [01/29/2025 19:25:56.574] [test-pekko.remote.default-remote-dispatcher-6] [pekko://test@10.30.170.247:17355/system/remoting-terminator] Remoting shut down. 07:25:56,574 PM [test-pekko.actor.default-dispatcher-18] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [DEBUG] [01/29/2025 19:25:56.574] [test-pekko.actor.internal-dispatcher-2] [EventStream] shutting down: StandardOutLogger