[DEBUG] [11/02/2024 23:15:57.829] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 11:15:57,829 PM [test-pekko.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [11/02/2024 23:15:57.829] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [11/02/2024 23:15:57.830] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [11/02/2024 23:15:57.837] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.171:17355] with UID [-7781375889683630159] [INFO] [11/02/2024 23:15:57.838] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.171:17355] - Starting up, Pekko version [1.0.3] ... 11:15:57,838 PM [test-pekko.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.171:17355] with UID [-7781375889683630159] 11:15:57,838 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.171:17355] - Starting up, Pekko version [1.0.3] ... 11:15:57,842 PM [test-pekko.actor.default-dispatcher-7] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.171:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 11:15:57,843 PM [test-pekko.actor.default-dispatcher-7] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.171:17355] - Started up successfully 11:15:57,843 PM [test-pekko.actor.default-dispatcher-7] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.171:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 11:15:57,843 PM [test-pekko.actor.default-dispatcher-7] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.171:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining [INFO] [11/02/2024 23:15:57.842] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.171:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [11/02/2024 23:15:57.842] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.171:17355] - Started up successfully [INFO] [11/02/2024 23:15:57.843] [test-pekko.actor.internal-dispatcher-5] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.171:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [11/02/2024 23:15:57.843] [test-pekko.actor.internal-dispatcher-5] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.171:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 11:15:57,853 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 11:15:57,853 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [11/02/2024 23:15:57.856] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower551]: org.opendaylight.controller.cluster.common.actor.Monitor@4752990a [INFO] [pekkoDeadLetter][11/02/2024 23:15:57.856] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower551] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:57,856 PM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower551] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:57,859 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower551 with journal batch size 1000 11:15:57,860 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: SnapshotOffer called. 11:15:57,860 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Moving last applied index from -1 to -1 11:15:57,860 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=1, votedFor=null 11:15:57,860 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING]] 11:15:57,860 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower551 in 74.68 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:15:57,860 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Recovery completed in 642.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:15:57,861 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower552 with journal batch size 1000 11:15:57,861 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: SnapshotOffer called. 11:15:57,861 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Moving last applied index from -1 to -1 11:15:57,861 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=1, votedFor=null 11:15:57,862 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 11:15:57,862 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower552 in 48.77 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:15:57,862 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Recovery completed in 447.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:15:57,863 PM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower552] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:57,863 PM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower553] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:57,864 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower553 with journal batch size 1000 [WARN] [11/02/2024 23:15:57.867] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower552]: org.opendaylight.controller.cluster.common.actor.Monitor@6736263c [INFO] [pekkoDeadLetter][11/02/2024 23:15:57.863] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower552] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [11/02/2024 23:15:57.867] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower553]: org.opendaylight.controller.cluster.common.actor.Monitor@290fea13 [INFO] [pekkoDeadLetter][11/02/2024 23:15:57.863] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower553] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:57,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: SnapshotOffer called. 11:15:57,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Moving last applied index from -1 to -1 11:15:57,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Set currentTerm=1, votedFor=null 11:15:57,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Updated server config: isVoting: false, peers: [PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=pekko://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 11:15:57,867 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower553 in 81.41 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [11/02/2024 23:15:57.867] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader550]: org.opendaylight.controller.cluster.common.actor.Monitor@359d8714 11:15:57,867 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Recovery completed in 597.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [INFO] [pekkoDeadLetter][11/02/2024 23:15:57.868] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader550] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:57,868 PM [test-pekko.actor.default-dispatcher-7] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader550] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:57,868 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader550 with journal batch size 1000 11:15:57,869 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: SnapshotOffer called. 11:15:57,869 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Moving last applied index from -1 to -1 11:15:57,869 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Set currentTerm=1, votedFor=null 11:15:57,869 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=pekko://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] 11:15:57,869 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader550 in 52.45 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:15:57,869 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Recovery completed in 451.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:15:58,257 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:15:58,257 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.171:17355, Removed) 11:15:58,259 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Received ElectionTimeout - switching to Candidate 11:15:58,259 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Election: Candidate has following voting peers: [follower552, follower551] 11:15:58,259 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Set currentTerm=2, votedFor=leader550 11:15:58,259 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Starting new election term 2 11:15:58,259 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] to peer follower552 11:15:58,259 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] to peer follower551 11:15:58,259 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:15:58,259 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:15:58,259 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Cluster state: [] 11:15:58,259 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:15:58,259 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:15:58,259 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=2, votedFor=null 11:15:58,260 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): In requestVote: RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:15:58,260 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=2, votedFor=leader550 11:15:58,260 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:15:58,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:15:58,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Cluster state: [] 11:15:58,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:15:58,260 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:15:58,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=2, votedFor=null 11:15:58,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): In requestVote: RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:15:58,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=2, votedFor=leader550 11:15:58,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:15:58,260 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:15:58,260 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:15:58,261 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Election: Leader has following peers: [follower553, follower552, follower551] 11:15:58,261 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:58,261 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:58,261 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:58,261 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553 (Follower): Term 2 in "AppendEntries [leaderId=leader550, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:15:58,261 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Set currentTerm=2, votedFor=null 11:15:58,261 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:15:58,261 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower553: Last sync leader does not match current leader leader550, need to catch up to -1 11:15:58,261 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower553 updated: matchIndex: -1, nextIndex: 0 11:15:58,261 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:58,262 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower551: Last sync leader does not match current leader leader550, need to catch up to -1 11:15:58,262 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower552: Last sync leader does not match current leader leader550, need to catch up to -1 11:15:58,262 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower551 updated: matchIndex: -1, nextIndex: 0 11:15:58,262 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:58,262 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower552 updated: matchIndex: -1, nextIndex: 0 [WARN] [11/02/2024 23:15:58.262] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower553] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@73491e20 11:15:58,262 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:15:58.262] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower552] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@364b9e2d [INFO] [pekkoDeadLetter][11/02/2024 23:15:58.262] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][11/02/2024 23:15:58.262] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:58,262 PM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:58,262 PM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:58,263 PM [test-pekko.actor.default-dispatcher-11] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [11/02/2024 23:15:58.263] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower551] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@75ad7e5d [INFO] [pekkoDeadLetter][11/02/2024 23:15:58.263] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:58,278 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:15:58,278 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader550 starting 11:15:58,280 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: onShutDown 11:15:58,280 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Initiating leader transfer 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader550: pauseLeader successfully completed - doing transfer 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Attempting to transfer leadership 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,281 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] [WARN] [11/02/2024 23:15:58.282] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower553] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7b3ffd26 [WARN] [11/02/2024 23:15:58.282] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower552] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5232723f [WARN] [11/02/2024 23:15:58.282] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower551] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@41d4a73e [INFO] [pekkoDeadLetter][11/02/2024 23:15:58.282] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:58,282 PM [test-pekko.actor.default-dispatcher-12] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower553] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][11/02/2024 23:15:58.282] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:58,282 PM [test-pekko.actor.default-dispatcher-12] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][11/02/2024 23:15:58.283] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:58,283 PM [test-pekko.actor.default-dispatcher-12] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:15:58,377 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,377 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,377 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,497 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,497 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,497 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,617 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader550: leader transfer aborted 11:15:58,617 PM [test-pekko.actor.default-dispatcher-17] [WARN] RaftActorLeadershipTransferCohort - leader550: Failed to transfer leadership in 336.1 ms 11:15:58,617 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: leader transfer failed - sending PoisonPill 11:15:58,617 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Leadership transfer expired 11:15:58,617 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,617 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 11:15:58,617 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [11/02/2024 23:15:58.617] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/leader550] received dead letter from TestActor[pekko://test/user/follower552]: AppendEntriesReply [term=2, success=true, followerId=follower552, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [11/02/2024 23:15:58.618] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/leader550] received dead letter from TestActor[pekko://test/user/follower551]: AppendEntriesReply [term=2, success=true, followerId=follower551, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:15:58,618 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader550 ending 11:15:58,619 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 11:15:58,619 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower551-notifier554#-1527959968] 11:15:58,630 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower551-collector555#1541904489] 11:15:58,630 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower551] 11:15:58,631 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower552-notifier556#-1618692848] 11:15:58,631 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower552-collector557#-1946338380] 11:15:58,631 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower552] 11:15:58,631 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower553-notifier558#-1262008954] 11:15:58,631 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower553-collector559#-673615833] 11:15:58,631 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower553] 11:15:58,631 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader550-notifier560#1043210950] 11:15:58,631 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader550-collector561#1793950329] 11:15:58,631 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader550] 11:15:58,633 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 11:15:58,633 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [11/02/2024 23:15:58.636] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower563]: org.opendaylight.controller.cluster.common.actor.Monitor@2d3f347f 11:15:58,637 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower563 with journal batch size 1000 11:15:58,637 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: SnapshotOffer called. 11:15:58,637 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Moving last applied index from -1 to -1 11:15:58,637 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=1, votedFor=null 11:15:58,637 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 11:15:58,637 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower563 in 72.34 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:15:58,638 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Recovery completed in 828.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [11/02/2024 23:15:58.639] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower564]: org.opendaylight.controller.cluster.common.actor.Monitor@6c5978df 11:15:58,640 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower564 with journal batch size 1000 11:15:58,641 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: SnapshotOffer called. 11:15:58,641 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Moving last applied index from -1 to -1 11:15:58,641 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=1, votedFor=null 11:15:58,641 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 11:15:58,641 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower564 in 38.32 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:15:58,641 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Recovery completed in 336.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:15:58,642 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower565 with journal batch size 1000 11:15:58,642 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: SnapshotOffer called. 11:15:58,642 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Moving last applied index from -1 to -1 11:15:58,642 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=1, votedFor=null 11:15:58,642 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Updated server config: isVoting: false, peers: [PeerInfo [id=leader562, address=pekko://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING]] 11:15:58,642 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower565 in 35.85 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:15:58,643 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader562 with journal batch size 1000 11:15:58,643 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: SnapshotOffer called. 11:15:58,643 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Moving last applied index from -1 to -1 11:15:58,643 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=1, votedFor=null 11:15:58,643 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Updated server config: isVoting: true, peers: [PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=pekko://test/user/follower565, votingState=NON_VOTING]] 11:15:58,643 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader562 in 34.46 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [11/02/2024 23:15:58.643] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower565]: org.opendaylight.controller.cluster.common.actor.Monitor@3fb80f48 [WARN] [11/02/2024 23:15:58.643] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader562]: org.opendaylight.controller.cluster.common.actor.Monitor@1bdf55da 11:15:58,643 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Recovery completed in 454.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:15:58,643 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Recovery completed in 364.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:15:59,6 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:15:59,7 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.171:17355, Removed) 11:15:59,7 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Received ElectionTimeout - switching to Candidate 11:15:59,7 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Election: Candidate has following voting peers: [follower564, follower563] 11:15:59,7 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=2, votedFor=leader562 11:15:59,7 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Starting new election term 2 11:15:59,7 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] to peer follower564 11:15:59,7 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] to peer follower563 11:15:59,7 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:15:59,7 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:15:59,7 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 11:15:59,7 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:15:59,7 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:15:59,8 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=2, votedFor=null 11:15:59,8 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:15:59,8 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=2, votedFor=leader562 11:15:59,8 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:15:59,9 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:15:59,8 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:15:59,9 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:15:59,9 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Cluster state: [] 11:15:59,9 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:15:59,9 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:15:59,9 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=2, votedFor=null 11:15:59,9 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): In requestVote: RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:15:59,9 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Election: Leader has following peers: [follower564, follower563, follower565] 11:15:59,9 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=2, votedFor=leader562 11:15:59,9 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:15:59,9 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,9 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,9 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,9 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 2 in "AppendEntries [leaderId=leader562, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:15:59,9 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=2, votedFor=null 11:15:59,9 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader leader562, need to catch up to -1 11:15:59,9 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:15:59,10 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 11:15:59,10 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,10 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader leader562, need to catch up to -1 11:15:59,10 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 11:15:59,10 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:15:59.010] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@816eefe 11:15:59,11 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower564: Last sync leader does not match current leader leader562, need to catch up to -1 11:15:59,11 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower564 updated: matchIndex: -1, nextIndex: 0 [WARN] [11/02/2024 23:15:59.011] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5bfe7632 11:15:59,11 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:15:59.011] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@b47d59b 11:15:59,50 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:15:59,50 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:15:59,50 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:15:59,50 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:15:59,50 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:15:59,51 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: onRequestLeadership RequestLeadership [requestedFollowerId=follower564, replyTo=Actor[pekko://test/user/$a#845421359]] 11:15:59,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Initiating leader transfer 11:15:59,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader562: pauseLeader successfully completed - doing transfer 11:15:59,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Attempting to transfer leadership 11:15:59,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): tryToCompleteLeadershipTransfer: followerId: follower564, matchIndex: -1, lastIndex: -1, isVoting: true 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Follower's log matches - sending ElectionTimeout 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leader transfer complete 11:15:59,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader transfer complete - waiting for new leader [WARN] [11/02/2024 23:15:59.054] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3dd3e4fd [WARN] [11/02/2024 23:15:59.054] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1bee8e43 [WARN] [11/02/2024 23:15:59.054] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7cb60ba0 11:15:59,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 11:15:59,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:15:59,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Received TimeoutNow - switching to Candidate 11:15:59,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Election: Candidate has following voting peers: [leader562, follower563] 11:15:59,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=3, votedFor=follower564 11:15:59,54 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Starting new election term 3 11:15:59,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] to peer leader562 11:15:59,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] to peer follower563 11:15:59,54 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 2 - updating term 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=3, votedFor=null 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=3, votedFor=follower564 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 11:15:59,55 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 11:15:59,55 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 11:15:59,55 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:15:59,55 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Election: Leader has following peers: [leader562, follower563, follower565] 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Cluster state: [] 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:15:59,55 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 11:15:59,55 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=3, votedFor=null 11:15:59,55 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,55 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 11:15:59,55 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 3 in "AppendEntries [leaderId=follower564, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 11:15:59,55 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=3, votedFor=null 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leadership transfer in progress - processing RequestVote 11:15:59,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): In requestVote: RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:15:59,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=3, votedFor=follower564 11:15:59,56 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader follower564, need to catch up to -1 11:15:59,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 11:15:59,56 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader follower564, need to catch up to -1 11:15:59,56 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader) :- Switching from behavior Leader to Follower, election term: 3 [WARN] [11/02/2024 23:15:59.056] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@62a89fa6 11:15:59,56 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 11:15:59,56 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,56 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 11:15:59,56 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:15:59.056] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower565] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2c436711 11:15:59,56 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - leader562: Last sync leader does not match current leader follower564, need to catch up to -1 11:15:59,56 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader changed to follower564 11:15:59,56 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorLeadershipTransferCohort - leader562: Successfully transferred leadership to follower564 in 4.248 ms 11:15:59,57 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Leadership transferred successfully to follower564 11:15:59,57 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader562 updated: matchIndex: -1, nextIndex: 0 11:15:59,57 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:15:59.057] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader562] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6ea89b45 11:15:59,102 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 11:15:59,102 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower563-notifier566#1018124053] 11:15:59,113 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower563-collector567#-819478662] 11:15:59,113 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower563] 11:15:59,113 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower564-notifier568#16236912] 11:15:59,113 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower564-collector569#1870431747] 11:15:59,113 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower564] 11:15:59,114 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower565-notifier570#-1394436929] 11:15:59,114 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower565-collector571#-968270541] 11:15:59,114 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower565] 11:15:59,114 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader562-notifier572#1935347140] 11:15:59,114 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader562-collector573#-1726411668] 11:15:59,114 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader562] 11:15:59,114 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$a#845421359] 11:15:59,116 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 11:15:59,116 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [11/02/2024 23:15:59.117] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower575]: org.opendaylight.controller.cluster.common.actor.Monitor@20c0303b 11:15:59,118 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower575 with journal batch size 1000 11:15:59,119 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: SnapshotOffer called. 11:15:59,119 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Moving last applied index from -1 to -1 11:15:59,119 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=1, votedFor=null 11:15:59,119 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 11:15:59,119 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower575 in 92.15 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:15:59,120 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Recovery completed in 878.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [11/02/2024 23:15:59.120] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower576]: org.opendaylight.controller.cluster.common.actor.Monitor@1014a7c3 11:15:59,120 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower576 with journal batch size 1000 11:15:59,121 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: SnapshotOffer called. 11:15:59,121 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Moving last applied index from -1 to -1 11:15:59,121 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=1, votedFor=null 11:15:59,121 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING]] 11:15:59,121 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower576 in 41.73 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:15:59,122 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Recovery completed in 602.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [11/02/2024 23:15:59.221] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower577]: org.opendaylight.controller.cluster.common.actor.Monitor@18d0b5b6 11:15:59,221 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower577 with journal batch size 1000 11:15:59,222 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: SnapshotOffer called. 11:15:59,222 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Moving last applied index from -1 to -1 11:15:59,222 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Set currentTerm=1, votedFor=null 11:15:59,222 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Updated server config: isVoting: false, peers: [PeerInfo [id=leader574, address=pekko://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 11:15:59,222 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower577 in 213.4 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [11/02/2024 23:15:59.223] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader574]: org.opendaylight.controller.cluster.common.actor.Monitor@79f7eaee 11:15:59,223 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Recovery completed in 1.101 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:15:59,224 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader574 with journal batch size 1000 11:15:59,224 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: SnapshotOffer called. 11:15:59,224 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Moving last applied index from -1 to -1 11:15:59,224 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Set currentTerm=1, votedFor=null 11:15:59,224 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Updated server config: isVoting: true, peers: [PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=pekko://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=VOTING]] 11:15:59,224 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader574 in 117.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:15:59,225 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Recovery completed in 708.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:15:59,617 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:15:59,617 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.171:17355, Removed) 11:15:59,617 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Received ElectionTimeout - switching to Candidate 11:15:59,617 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Election: Candidate has following voting peers: [follower575, follower576] 11:15:59,617 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Set currentTerm=2, votedFor=leader574 11:15:59,617 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Starting new election term 2 11:15:59,617 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] to peer follower575 11:15:59,617 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] to peer follower576 11:15:59,617 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:15:59,617 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:15:59,617 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Cluster state: [] 11:15:59,617 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:15:59,617 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:15:59,617 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:15:59,617 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=2, votedFor=null 11:15:59,617 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Cluster state: [] 11:15:59,617 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:15:59,617 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:15:59,617 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=2, votedFor=null 11:15:59,617 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): In requestVote: RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:15:59,618 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=2, votedFor=leader574 11:15:59,618 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:15:59,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): In requestVote: RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:15:59,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=2, votedFor=leader574 11:15:59,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:15:59,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:15:59,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:15:59,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Election: Leader has following peers: [follower575, follower577, follower576] 11:15:59,619 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,619 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,619 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,619 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:15:59,619 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577 (Follower): Term 2 in "AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:15:59,619 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Set currentTerm=2, votedFor=null 11:15:59,619 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower577: Last sync leader does not match current leader leader574, need to catch up to -1 11:15:59,619 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower577 updated: matchIndex: -1, nextIndex: 0 11:15:59,619 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,619 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower576: Last sync leader does not match current leader leader574, need to catch up to -1 11:15:59,619 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower576 updated: matchIndex: -1, nextIndex: 0 [WARN] [11/02/2024 23:15:59.619] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower576] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@27751ab 11:15:59,619 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,619 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower575: Last sync leader does not match current leader leader574, need to catch up to -1 11:15:59,619 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower575 updated: matchIndex: -1, nextIndex: 0 [WARN] [11/02/2024 23:15:59.619] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower577] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@22abcca5 11:15:59,620 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:15:59.620] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower575] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7b55b014 11:15:59,629 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:15:59,629 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:15:59,629 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:15:59,629 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower576 starting 11:15:59,629 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: onShutDown 11:15:59,630 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower576 ending 11:15:59,630 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:15:59,630 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: onRequestLeadership RequestLeadership [requestedFollowerId=follower576, replyTo=Actor[pekko://test/user/$b#-1634955226]] 11:15:59,630 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Initiating leader transfer 11:15:59,630 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] 11:15:59,630 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader574: pauseLeader successfully completed - doing transfer 11:15:59,630 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Attempting to transfer leadership 11:15:59,630 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] 11:15:59,630 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,630 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,630 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:15:59,630 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 11:15:59,630 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 11:15:59,630 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending [WARN] [11/02/2024 23:15:59.630] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: LeaderTransitioning [leaderId=leader574] [WARN] [11/02/2024 23:15:59.630] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [11/02/2024 23:15:59.630] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower575] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7723b93a [WARN] [11/02/2024 23:15:59.630] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower577] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3e178b85 11:15:59,737 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 11:15:59,737 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [11/02/2024 23:15:59.737] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:15:59,856 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 11:15:59,857 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [11/02/2024 23:15:59.857] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:15:59,976 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader574: leader transfer aborted 11:15:59,976 PM [test-pekko.actor.default-dispatcher-16] [WARN] RaftActorLeadershipTransferCohort - leader574: Failed to transfer leadership in 346.7 ms 11:15:59,977 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: LeadershipTransfer request from follower576 failed 11:15:59,977 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Leadership transfer expired 11:15:59,977 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 11:15:59,978 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [11/02/2024 23:15:59.978] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower576] received dead letter from TestActor[pekko://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:15:59,983 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 11:15:59,983 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower575-notifier578#-638060251] 11:15:59,994 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower575-collector579#-607061532] 11:15:59,994 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower575] 11:15:59,995 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower576-notifier580#-1073447143] 11:15:59,995 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower576-collector581#-1296087203] 11:15:59,995 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower576] 11:15:59,995 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower577-notifier582#-1417669775] 11:15:59,995 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower577-collector583#2097239798] 11:15:59,995 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower577] 11:15:59,995 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader574-notifier584#-36665557] 11:15:59,995 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader574-collector585#-1308286477] 11:15:59,995 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader574] 11:15:59,996 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$b#-1634955226] 11:15:59,996 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 11:15:59,996 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [11/02/2024 23:16:00.000] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower587]: org.opendaylight.controller.cluster.common.actor.Monitor@1e1ad3d7 11:16:00,1 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower587 with journal batch size 1000 11:16:00,1 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: SnapshotOffer called. 11:16:00,1 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to -1 11:16:00,1 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=1, votedFor=null 11:16:00,1 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING]] 11:16:00,1 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower587 in 65.57 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [11/02/2024 23:16:00.002] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower588]: org.opendaylight.controller.cluster.common.actor.Monitor@128a5d4c 11:16:00,2 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower588 with journal batch size 1000 11:16:00,2 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: SnapshotOffer called. 11:16:00,2 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Recovery completed in 957.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:00,2 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to -1 11:16:00,2 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=1, votedFor=null 11:16:00,3 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 11:16:00,3 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower588 in 57.55 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:00,3 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Recovery completed in 544.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [11/02/2024 23:16:00.004] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower589]: org.opendaylight.controller.cluster.common.actor.Monitor@16b09c56 11:16:00,4 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower589 with journal batch size 1000 11:16:00,5 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: SnapshotOffer called. 11:16:00,5 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to -1 11:16:00,5 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=1, votedFor=null 11:16:00,5 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Updated server config: isVoting: false, peers: [PeerInfo [id=leader586, address=pekko://test/user/leader586, votingState=VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 11:16:00,5 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower589 in 62.73 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:00,5 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Recovery completed in 542.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [11/02/2024 23:16:00.106] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader586]: org.opendaylight.controller.cluster.common.actor.Monitor@3d6aca18 11:16:00,106 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader586 with journal batch size 1000 11:16:00,107 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: SnapshotOffer called. 11:16:00,107 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to -1 11:16:00,107 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=1, votedFor=null 11:16:00,107 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Updated server config: isVoting: true, peers: [PeerInfo [id=follower589, address=pekko://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 11:16:00,107 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader586 in 74.07 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:00,108 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Recovery completed in 819.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:00,437 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:00,437 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.171:17355, Removed) 11:16:00,437 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Received ElectionTimeout - switching to Candidate 11:16:00,437 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Election: Candidate has following voting peers: [follower588, follower587] 11:16:00,437 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=2, votedFor=leader586 11:16:00,437 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Starting new election term 2 11:16:00,437 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] to peer follower588 11:16:00,437 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] to peer follower587 11:16:00,437 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:16:00,437 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:00,438 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:00,438 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Cluster state: [] 11:16:00,438 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 11:16:00,438 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:16:00,438 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:16:00,438 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:16:00,438 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:16:00,438 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=2, votedFor=null 11:16:00,438 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=2, votedFor=null 11:16:00,438 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): In requestVote: RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:00,438 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:00,438 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=2, votedFor=leader586 11:16:00,438 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=2, votedFor=leader586 11:16:00,438 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:16:00,438 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:16:00,438 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:16:00,438 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:16:00,439 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Election: Leader has following peers: [follower589, follower588, follower587] 11:16:00,439 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:00,439 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:00,439 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:00,439 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 2 in "AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:16:00,439 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=2, votedFor=null 11:16:00,439 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:16:00,439 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader leader586, need to catch up to -1 11:16:00,439 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower587: Last sync leader does not match current leader leader586, need to catch up to -1 11:16:00,439 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: -1, nextIndex: 0 11:16:00,439 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:00,439 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: -1, nextIndex: 0 11:16:00,439 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:16:00.440] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@73d04577 11:16:00,440 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader leader586, need to catch up to -1 11:16:00,440 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: -1, nextIndex: 0 11:16:00,440 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:16:00.440] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@23ddc80c [WARN] [11/02/2024 23:16:00.441] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@12b3b32 11:16:00,463 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:16:00,463 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 11:16:00,463 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:00,463 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:00,463 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 11:16:00,463 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,463 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower589 11:16:00,464 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower589: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:00,464 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:00,464 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Number of entries to be appended = 1 11:16:00,464 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:16:00,464 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Append entry to log zero 11:16:00,464 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:00,464 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Log size is now 1 11:16:00,464 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower589, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:16:00,464 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 11:16:00,464 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,464 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 11:16:00,464 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:00,464 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,465 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower587 11:16:00,465 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [11/02/2024 23:16:00.465] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3e299b9b 11:16:00,465 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower587: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:00,465 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:00,465 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Number of entries to be appended = 1 11:16:00,465 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:16:00,465 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Append entry to log zero 11:16:00,465 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:00,465 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Log size is now 1 11:16:00,465 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower587, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:16:00,465 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 11:16:00,465 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 11:16:00,465 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,466 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: 0, nextIndex: 1 11:16:00,466 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,466 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [11/02/2024 23:16:00.466] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@60955b4d 11:16:00,466 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:00,466 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 11:16:00,467 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Setting last applied to 0 11:16:00,467 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to 0 11:16:00,467 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Applying state for log index 0 data zero 11:16:00,467 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: applyState called: zero 11:16:00,467 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persisting ApplyJournalEntries with index=0 11:16:00,556 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 11:16:00,556 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:00,677 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 11:16:00,677 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:00,677 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Commit index set to 0 11:16:00,677 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 11:16:00,677 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:16:00,677 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Setting last applied to 0 11:16:00,677 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to 0 11:16:00,677 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Applying state for log index 0 data zero 11:16:00,677 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: applyState called: zero 11:16:00,677 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Persisting ApplyJournalEntries with index=0 11:16:00,678 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Commit index set to 0 11:16:00,678 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 11:16:00,678 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:16:00,678 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Setting last applied to 0 11:16:00,678 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to 0 11:16:00,678 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Applying state for log index 0 data zero 11:16:00,678 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: applyState called: zero 11:16:00,679 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Persisting ApplyJournalEntries with index=0 11:16:00,715 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 11:16:00,715 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: onShutDown 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Initiating leader transfer 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader586: pauseLeader successfully completed - doing transfer 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Attempting to transfer leadership 11:16:00,716 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,716 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 11:16:00,716 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower589, matchIndex: 0, lastIndex: 0, isVoting: false 11:16:00,716 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower587, matchIndex: 0, lastIndex: 0, isVoting: true 11:16:00,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Follower's log matches - sending ElectionTimeout 11:16:00,717 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,717 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,717 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 11:16:00,717 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,717 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leader transfer complete 11:16:00,717 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader transfer complete - waiting for new leader 11:16:00,717 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 11:16:00,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 11:16:00,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:00,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Received TimeoutNow - switching to Candidate 11:16:00,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Election: Candidate has following voting peers: [leader586, follower588] 11:16:00,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=3, votedFor=follower587 11:16:00,717 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Starting new election term 3 11:16:00,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] to peer leader586 11:16:00,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] to peer follower588 11:16:00,717 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:00,718 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 11:16:00,718 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Cluster state: [] 11:16:00,718 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:00,718 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 11:16:00,718 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2]" message is greater than leader's term 2 - switching to Follower 11:16:00,718 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=3, votedFor=null 11:16:00,718 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 11:16:00,718 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 11:16:00,718 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2]" message is greater than follower's term 2 - updating term 11:16:00,718 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=3, votedFor=null 11:16:00,718 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leadership transfer in progress - processing RequestVote 11:16:00,718 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): In requestVote: RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 11:16:00,718 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=3, votedFor=follower587 11:16:00,718 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:00,718 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 11:16:00,718 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=3, votedFor=follower587 11:16:00,718 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 11:16:00,718 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 11:16:00,718 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 11:16:00,719 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader) :- Switching from behavior Leader to Follower, election term: 3 11:16:00,719 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Election: Leader has following peers: [leader586, follower589, follower588] 11:16:00,719 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,719 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,719 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,719 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 3 in "AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 11:16:00,719 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 11:16:00,719 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=3, votedFor=null 11:16:00,719 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader follower587, need to catch up to -1 11:16:00,719 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 11:16:00,719 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,720 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - leader586: Last sync leader does not match current leader follower587, need to catch up to -1 11:16:00,720 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader586 updated: matchIndex: 0, nextIndex: 1 11:16:00,720 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader changed to follower587 11:16:00,720 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:00,720 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorLeadershipTransferCohort - leader586: Successfully transferred leadership to follower587 in 4.246 ms [WARN] [11/02/2024 23:16:00.720] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@37dea17c 11:16:00,720 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 11:16:00,720 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: leader transfer succeeded - sending PoisonPill 11:16:00,720 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - leader586: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) [WARN] [11/02/2024 23:16:00.720] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower589] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@26406b6 [WARN] [11/02/2024 23:16:00.721] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/leader586] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@75dd05a6 [WARN] [11/02/2024 23:16:00.721] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/leader586] received dead letter: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@23237899 [WARN] [11/02/2024 23:16:01.237] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader586] received dead letter from TestActor[pekko://test/user/follower587]: AppendEntries [leaderId=follower587, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:16:01,237 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader follower587, need to catch up to -1 11:16:01,237 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): sendAppendEntries: 0 is present for follower follower588 11:16:01,237 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries: AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Number of entries to be appended = 1 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Append entry to log zero 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Log size is now 1 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Commit index set to 0 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries returning : AppendEntriesReply [term=3, success=true, followerId=follower588, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Setting last applied to 0 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to 0 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Applying state for log index 0 data zero 11:16:01,237 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: 0, nextIndex: 1 11:16:01,237 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: applyState called: zero 11:16:01,238 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:01,238 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:01,239 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Persisting ApplyJournalEntries with index=0 [WARN] [11/02/2024 23:16:01.239] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@170da2fc [WARN] [11/02/2024 23:16:01.239] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@76c0326b 11:16:01,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 11:16:01,273 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 11:16:01,273 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower588 starting 11:16:01,273 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onShutDown 11:16:01,274 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower588 ending 11:16:01,274 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 11:16:01,274 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587-notifier590#-808014841] 11:16:01,285 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587-collector591#-1586649220] 11:16:01,285 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower587] 11:16:01,285 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower588-notifier592#-998892004] 11:16:01,286 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower588-collector593#-1267665937] 11:16:01,286 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower588] 11:16:01,286 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower589-notifier594#-259108737] 11:16:01,286 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower589-collector595#945945842] 11:16:01,286 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower589] 11:16:01,287 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader586-notifier596#1056626710] 11:16:01,287 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader586-collector597#-1849427146] 11:16:01,287 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader586] 11:16:01,288 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [11/02/2024 23:16:01.289] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader598]: org.opendaylight.controller.cluster.common.actor.Monitor@2bcd2b12 11:16:01,290 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader598 with journal batch size 1000 11:16:01,291 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:01,292 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:01,292 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower): Received TimeoutNow - switching to Candidate 11:16:01,292 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Election: Candidate has following voting peers: [] 11:16:01,292 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Set currentTerm=1, votedFor=leader598 11:16:01,292 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Starting new election term 1 11:16:01,292 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 11:16:01,293 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Received ElectionTimeout 11:16:01,293 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Leader): Election: Leader has following peers: [] 11:16:01,293 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 11:16:01,293 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader598 starting 11:16:01,293 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: onShutDown 11:16:01,294 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader598 ending 11:16:01,294 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 11:16:01,294 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader598-collector602#-1494500271] 11:16:01,305 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader598] 11:16:01,306 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 11:16:01,306 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [11/02/2024 23:16:01.311] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower604]: org.opendaylight.controller.cluster.common.actor.Monitor@6d3ac48d 11:16:01,312 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower604 with journal batch size 1000 11:16:01,313 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: SnapshotOffer called. 11:16:01,313 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to -1 11:16:01,313 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=1, votedFor=null 11:16:01,313 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 11:16:01,313 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower604 in 84.43 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:01,314 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Recovery completed in 800.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [11/02/2024 23:16:01.413] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower605]: org.opendaylight.controller.cluster.common.actor.Monitor@430fd844 11:16:01,414 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower605 with journal batch size 1000 [WARN] [11/02/2024 23:16:01.416] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower606]: org.opendaylight.controller.cluster.common.actor.Monitor@15852f13 11:16:01,416 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower606 with journal batch size 1000 11:16:01,417 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: SnapshotOffer called. 11:16:01,417 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to -1 11:16:01,417 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Set currentTerm=1, votedFor=null 11:16:01,417 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Updated server config: isVoting: false, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 11:16:01,417 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower606 in 78.57 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:01,418 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Recovery completed in 796.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:01,418 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: SnapshotOffer called. 11:16:01,418 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Moving last applied index from -1 to -1 11:16:01,418 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=1, votedFor=null 11:16:01,418 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Updated server config: isVoting: true, peers: [PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=pekko://test/user/leader603, votingState=VOTING]] 11:16:01,418 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower605 in 110.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:01,419 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Recovery completed in 1.331 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [11/02/2024 23:16:01.420] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader603]: org.opendaylight.controller.cluster.common.actor.Monitor@50dc22da 11:16:01,420 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader603 with journal batch size 1000 11:16:01,421 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: SnapshotOffer called. 11:16:01,421 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to -1 11:16:01,421 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Set currentTerm=1, votedFor=null 11:16:01,421 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=pekko://test/user/follower606, votingState=NON_VOTING]] 11:16:01,421 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader603 in 65.35 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:01,421 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Recovery completed in 674.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:01,826 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:01,826 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.171:17355, Removed) 11:16:01,826 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Received ElectionTimeout - switching to Candidate 11:16:01,826 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Election: Candidate has following voting peers: [follower605, follower604] 11:16:01,826 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Set currentTerm=2, votedFor=leader603 11:16:01,826 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Starting new election term 2 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] to peer follower605 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] to peer follower604 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Cluster state: [] 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=2, votedFor=null 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): In requestVote: RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=2, votedFor=leader603 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Cluster state: [] 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=2, votedFor=null 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): In requestVote: RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=2, votedFor=leader603 11:16:01,827 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:16:01,828 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:16:01,828 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:16:01,828 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Election: Leader has following peers: [follower605, follower604, follower606] 11:16:01,828 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:01,829 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:01,829 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:01,829 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower604: Last sync leader does not match current leader leader603, need to catch up to -1 11:16:01,829 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Term 2 in "AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:16:01,829 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Set currentTerm=2, votedFor=null 11:16:01,829 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower606: Last sync leader does not match current leader leader603, need to catch up to -1 [WARN] [11/02/2024 23:16:01.829] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower604] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6c7a37ba 11:16:01,829 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower605: Last sync leader does not match current leader leader603, need to catch up to -1 [WARN] [11/02/2024 23:16:01.829] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower605] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4251b948 11:16:01,829 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 [WARN] [11/02/2024 23:16:01.829] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower606] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1d4fa176 11:16:01,830 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: -1, nextIndex: 0 11:16:01,830 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:01,830 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: -1, nextIndex: 0 11:16:01,830 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:01,830 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower605 updated: matchIndex: -1, nextIndex: 0 11:16:01,830 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:01,878 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:16:01,878 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:16:01,878 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:16:01,878 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 11:16:01,878 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower604 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower604: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower606 11:16:01,879 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower606: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Number of entries to be appended = 1 11:16:01,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:16:01,879 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Append entry to log zero 11:16:01,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Number of entries to be appended = 1 11:16:01,879 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:16:01,879 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Append entry to log zero 11:16:01,879 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Log size is now 1 11:16:01,879 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Log size is now 1 11:16:01,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower604, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower606, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:16:01,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 11:16:01,880 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 11:16:01,880 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: 0, nextIndex: 1 11:16:01,880 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:01,880 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: 0, nextIndex: 1 11:16:01,880 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:01,881 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [11/02/2024 23:16:01.881] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower604] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6ed308b5 11:16:01,881 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [11/02/2024 23:16:01.881] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower606] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@46134701 11:16:01,881 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:16:01,881 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 11:16:01,881 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Setting last applied to 0 11:16:01,881 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to 0 11:16:01,881 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Applying state for log index 0 data zero 11:16:01,882 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: applyState called: zero 11:16:01,882 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persisting ApplyJournalEntries with index=0 11:16:01,947 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 11:16:01,947 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:02,67 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 11:16:02,67 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:16:02,67 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Commit index set to 0 11:16:02,67 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 11:16:02,67 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Commit index set to 0 11:16:02,67 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:16:02,67 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Setting last applied to 0 11:16:02,67 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 11:16:02,67 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to 0 11:16:02,67 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:16:02,67 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Applying state for log index 0 data zero 11:16:02,67 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Setting last applied to 0 11:16:02,67 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to 0 11:16:02,67 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Applying state for log index 0 data zero 11:16:02,67 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: applyState called: zero 11:16:02,67 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: applyState called: zero 11:16:02,67 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Persisting ApplyJournalEntries with index=0 11:16:02,67 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Persisting ApplyJournalEntries with index=0 11:16:02,80 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 11:16:02,80 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:16:02,80 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:16:02,80 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: onRequestLeadership RequestLeadership [requestedFollowerId=follower605, replyTo=Actor[pekko://test/user/$c#-123181215]] 11:16:02,80 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Initiating leader transfer 11:16:02,80 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader603: pauseLeader successfully completed - doing transfer 11:16:02,80 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 11:16:02,80 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Attempting to transfer leadership 11:16:02,80 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 11:16:02,80 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:02,80 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 11:16:02,80 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 11:16:02,80 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:02,80 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:02,81 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 11:16:02,81 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 11:16:02,186 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 11:16:02,186 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 11:16:02,307 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 11:16:02,307 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 11:16:02,426 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader603: leader transfer aborted 11:16:02,426 PM [test-pekko.actor.default-dispatcher-6] [WARN] RaftActorLeadershipTransferCohort - leader603: Failed to transfer leadership in 346.5 ms 11:16:02,426 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: LeadershipTransfer request from follower605 failed 11:16:02,427 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Leadership transfer expired 11:16:02,427 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 11:16:02,427 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 11:16:02,433 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 11:16:02,433 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower604-notifier607#-1504038668] 11:16:02,444 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower604-collector608#-1238753704] 11:16:02,445 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower604] 11:16:02,445 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower605-notifier609#-1320066012] 11:16:02,445 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower605-collector610#173980299] 11:16:02,445 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower605] 11:16:02,445 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower606-notifier611#1361439089] 11:16:02,446 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower606-collector612#-155920193] 11:16:02,446 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower606] 11:16:02,446 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader603-notifier613#1731675064] 11:16:02,446 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader603-collector614#-1741958819] 11:16:02,446 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader603] 11:16:02,446 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$c#-123181215] 11:16:02,447 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 11:16:02,447 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [11/02/2024 23:16:02.449] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower616]: org.opendaylight.controller.cluster.common.actor.Monitor@d514a7a 11:16:02,449 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower616 with journal batch size 1000 11:16:02,450 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: SnapshotOffer called. 11:16:02,450 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Moving last applied index from -1 to -1 11:16:02,450 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=1, votedFor=null 11:16:02,450 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Updated server config: isVoting: true, peers: [PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 11:16:02,450 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower616 in 90.06 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:02,451 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Recovery completed in 1.049 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [11/02/2024 23:16:02.550] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower617]: org.opendaylight.controller.cluster.common.actor.Monitor@14410f5b 11:16:02,551 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower617 with journal batch size 1000 11:16:02,551 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: SnapshotOffer called. 11:16:02,551 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Moving last applied index from -1 to -1 11:16:02,551 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=1, votedFor=null 11:16:02,551 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 11:16:02,551 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower617 in 67.74 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:02,552 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Recovery completed in 686.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [11/02/2024 23:16:02.553] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower618]: org.opendaylight.controller.cluster.common.actor.Monitor@548240ff 11:16:02,553 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower618 with journal batch size 1000 11:16:02,553 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: SnapshotOffer called. 11:16:02,553 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Moving last applied index from -1 to -1 11:16:02,553 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Set currentTerm=1, votedFor=null 11:16:02,554 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Updated server config: isVoting: false, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=pekko://test/user/leader615, votingState=VOTING]] 11:16:02,554 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower618 in 61.77 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:02,554 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Recovery completed in 662.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [11/02/2024 23:16:02.656] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader615]: org.opendaylight.controller.cluster.common.actor.Monitor@58217631 11:16:02,656 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader615 with journal batch size 1000 11:16:02,657 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: SnapshotOffer called. 11:16:02,657 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Moving last applied index from -1 to -1 11:16:02,657 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=1, votedFor=null 11:16:02,657 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=pekko://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=VOTING]] 11:16:02,657 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader615 in 70.38 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:02,658 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Recovery completed in 790.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:03,47 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:03,47 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.171:17355, Removed) 11:16:03,47 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Received ElectionTimeout - switching to Candidate 11:16:03,47 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Election: Candidate has following voting peers: [follower616, follower617] 11:16:03,47 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=2, votedFor=leader615 11:16:03,47 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Starting new election term 2 11:16:03,47 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] to peer follower616 11:16:03,47 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] to peer follower617 11:16:03,47 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:16:03,48 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:03,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:03,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Cluster state: [] 11:16:03,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:16:03,48 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:16:03,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=2, votedFor=null 11:16:03,48 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Cluster state: [] 11:16:03,48 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:16:03,48 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:16:03,48 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=2, votedFor=null 11:16:03,48 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): In requestVote: RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:03,48 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=2, votedFor=leader615 11:16:03,48 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:16:03,49 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): In requestVote: RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:03,49 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=2, votedFor=leader615 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:16:03,49 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Election: Leader has following peers: [follower616, follower618, follower617] 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:16:03,49 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower616: Last sync leader does not match current leader leader615, need to catch up to -1 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower616 updated: matchIndex: -1, nextIndex: 0 11:16:03,49 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618 (Follower): Term 2 in "AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,49 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Set currentTerm=2, votedFor=null 11:16:03,49 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower617: Last sync leader does not match current leader leader615, need to catch up to -1 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower617 updated: matchIndex: -1, nextIndex: 0 [WARN] [11/02/2024 23:16:03.050] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower616] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5fb935b7 [WARN] [11/02/2024 23:16:03.050] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower617] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6330c52b 11:16:03,49 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower618: Last sync leader does not match current leader leader615, need to catch up to -1 11:16:03,49 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,52 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower618 updated: matchIndex: -1, nextIndex: 0 11:16:03,52 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:16:03.053] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower618] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@20995635 11:16:03,58 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:16:03,58 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:16:03,59 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:16:03,59 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower616] 11:16:03,70 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower618] 11:16:03,81 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:16:03,81 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:16:03,81 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: onRequestLeadership RequestLeadership [requestedFollowerId=follower617, replyTo=Actor[pekko://test/user/$d#1550126025]] 11:16:03,81 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Initiating leader transfer [WARN] [11/02/2024 23:16:03.081] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: LeaderTransitioning [leaderId=leader615] 11:16:03,81 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader615: pauseLeader successfully completed - doing transfer 11:16:03,81 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Attempting to transfer leadership [WARN] [11/02/2024 23:16:03.081] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: LeaderTransitioning [leaderId=leader615] 11:16:03,81 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: onLeaderTransitioning: LeaderTransitioning [leaderId=leader615] 11:16:03,81 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,81 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,82 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:16:03.081] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [11/02/2024 23:16:03.082] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:16:03,82 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 11:16:03,82 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): tryToCompleteLeadershipTransfer: followerId: follower617, matchIndex: -1, lastIndex: -1, isVoting: true 11:16:03,82 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Follower's log matches - sending ElectionTimeout 11:16:03,82 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [11/02/2024 23:16:03.082] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower617] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6596a05a [WARN] [11/02/2024 23:16:03.082] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:16:03,82 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,82 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,82 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leader transfer complete 11:16:03,82 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader transfer complete - waiting for new leader [WARN] [11/02/2024 23:16:03.082] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:16:03,82 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 11:16:03,83 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:03,83 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Received TimeoutNow - switching to Candidate 11:16:03,83 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Election: Candidate has following voting peers: [follower616, leader615] 11:16:03,83 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=3, votedFor=follower617 11:16:03,83 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Starting new election term 3 11:16:03,83 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] to peer follower616 11:16:03,83 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] to peer leader615 [WARN] [11/02/2024 23:16:03.083] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/follower617]: RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] 11:16:03,83 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 11:16:03,83 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:03,83 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Cluster state: [] 11:16:03,83 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:16:03,83 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 11:16:03,83 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=3, votedFor=null 11:16:03,83 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leadership transfer in progress - processing RequestVote 11:16:03,83 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): In requestVote: RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:03,83 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=3, votedFor=follower617 11:16:03,83 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 11:16:03,83 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader) :- Switching from behavior Leader to Follower, election term: 3 11:16:03,84 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 11:16:03,84 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 11:16:03,85 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Election: Leader has following peers: [follower616, follower618, leader615] 11:16:03,85 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,85 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,85 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,85 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [11/02/2024 23:16:03.085] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower616] received dead letter from TestActor[pekko://test/user/follower617]: AppendEntries [leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [11/02/2024 23:16:03.085] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower618] received dead letter from TestActor[pekko://test/user/follower617]: AppendEntries [leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:16:03,85 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - leader615: Last sync leader does not match current leader follower617, need to catch up to -1 11:16:03,85 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader changed to follower617 11:16:03,85 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader615 updated: matchIndex: -1, nextIndex: 0 11:16:03,85 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:03,85 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorLeadershipTransferCohort - leader615: Successfully transferred leadership to follower617 in 4.127 ms 11:16:03,85 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Leadership transferred successfully to follower617 [WARN] [11/02/2024 23:16:03.086] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader615] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@44b61f00 11:16:03,132 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 11:16:03,132 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower616-notifier619#1540401420] 11:16:03,143 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower616-collector620#-1102914172] 11:16:03,143 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower617-notifier621#-714826024] 11:16:03,143 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower617-collector622#-1992600419] 11:16:03,143 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower617] 11:16:03,143 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower618-notifier623#-969353186] 11:16:03,144 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower618-collector624#1886433545] 11:16:03,144 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader615-notifier625#-830829360] 11:16:03,144 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader615-collector626#637132607] 11:16:03,144 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader615] 11:16:03,144 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$d#1550126025] [INFO] [11/02/2024 23:16:03.145] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 11:16:03,146 PM [test-pekko.actor.default-dispatcher-18] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [11/02/2024 23:16:03.146] [test-pekko.actor.default-dispatcher-18] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.171.171:17355) [INFO] [11/02/2024 23:16:03.147] [test-pekko.actor.internal-dispatcher-5] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.171:17355] - Exiting completed 11:16:03,147 PM [test-pekko.actor.default-dispatcher-15] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.171:17355] - Exiting completed 11:16:03,147 PM [test-pekko.actor.default-dispatcher-16] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.171:17355] - Shutting down... [INFO] [11/02/2024 23:16:03.147] [test-pekko.actor.internal-dispatcher-5] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.171:17355] - Shutting down... 11:16:03,147 PM [test-pekko.actor.default-dispatcher-16] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.171:17355] - Successfully shut down [INFO] [11/02/2024 23:16:03.147] [test-pekko.actor.internal-dispatcher-5] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.171:17355] - Successfully shut down 11:16:03,149 PM [test-pekko.actor.default-dispatcher-16] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 11:16:03,149 PM [test-pekko.actor.default-dispatcher-16] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [11/02/2024 23:16:03.149] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.171.171:17355/system/remoting-terminator] Shutting down remote daemon. [INFO] [11/02/2024 23:16:03.149] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.171.171:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports.