[DEBUG] [02/27/2024 11:14:20.343] [main] [EventStream(akka://test)] logger log1-TestEventListener started 11:14:20,361 AM [test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [02/27/2024 11:14:20.361] [main] [EventStream(akka://test)] logger log2-Slf4jLogger started [DEBUG] [02/27/2024 11:14:20.361] [main] [EventStream(akka://test)] Default Loggers started [INFO] [02/27/2024 11:14:20.385] [main] [ArteryTransport(akka://test)] Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.171.238:25520] with UID [-6619412372108872494] [INFO] [02/27/2024 11:14:20.386] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.238:25520] - Starting up, Akka version [2.6.21] ... 11:14:20,388 AM [test-akka.actor.default-dispatcher-7] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.171.238:25520] with UID [-6619412372108872494] 11:14:20,388 AM [test-akka.actor.default-dispatcher-7] [INFO] Cluster - Cluster Node [akka://test@10.30.171.238:25520] - Starting up, Akka version [2.6.21] ... [INFO] [02/27/2024 11:14:20.391] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.238:25520] - Registered cluster JMX MBean [akka:type=Cluster] [INFO] [02/27/2024 11:14:20.391] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.238:25520] - Started up successfully 11:14:20,392 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.238:25520] - Registered cluster JMX MBean [akka:type=Cluster] 11:14:20,392 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.238:25520] - Started up successfully [INFO] [02/27/2024 11:14:20.394] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.238:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing [INFO] [02/27/2024 11:14:20.394] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.238:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 11:14:20,395 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.238:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 11:14:20,395 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.238:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 11:14:20,409 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 11:14:20,409 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [02/27/2024 11:14:20.414] [test-akka.actor.default-dispatcher-7] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower507]: org.opendaylight.controller.cluster.common.actor.Monitor@1b4ab44a [INFO] [akkaDeadLetter][02/27/2024 11:14:20.414] [test-akka.actor.default-dispatcher-7] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower507] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:20,416 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower507 with journal batch size 1000 11:14:20,417 AM [test-akka.actor.default-dispatcher-7] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower507] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [02/27/2024 11:14:20.418] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower508]: org.opendaylight.controller.cluster.common.actor.Monitor@19d9c545 [INFO] [akkaDeadLetter][02/27/2024 11:14:20.418] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower508] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:20,418 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower508] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:20,419 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower508 with journal batch size 1000 11:14:20,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: SnapshotOffer called. 11:14:20,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Moving last applied index from -1 to -1 11:14:20,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Set currentTerm=1, votedFor=null 11:14:20,420 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: SnapshotOffer called. 11:14:20,420 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Moving last applied index from -1 to -1 11:14:20,420 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Set currentTerm=1, votedFor=null 11:14:20,422 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Updated server config: isVoting: true, peers: [PeerInfo [id=follower507, address=akka://test/user/follower507, votingState=VOTING], PeerInfo [id=leader506, address=akka://test/user/leader506, votingState=VOTING], PeerInfo [id=follower509, address=akka://test/user/follower509, votingState=NON_VOTING]] 11:14:20,422 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower508 in 1.850 ms: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:20,422 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Updated server config: isVoting: true, peers: [PeerInfo [id=follower508, address=akka://test/user/follower508, votingState=VOTING], PeerInfo [id=leader506, address=akka://test/user/leader506, votingState=VOTING], PeerInfo [id=follower509, address=akka://test/user/follower509, votingState=NON_VOTING]] 11:14:20,423 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower507 in 3.135 ms: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:20,423 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Recovery completed in 3.005 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:20,424 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Recovery completed in 4.523 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:20.521] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower509]: org.opendaylight.controller.cluster.common.actor.Monitor@3328e8d9 11:14:20,521 AM [test-akka.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower509] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][02/27/2024 11:14:20.521] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower509] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:20,522 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower509 with journal batch size 1000 11:14:20,523 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: SnapshotOffer called. 11:14:20,523 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: Moving last applied index from -1 to -1 11:14:20,523 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: Set currentTerm=1, votedFor=null 11:14:20,523 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: Updated server config: isVoting: false, peers: [PeerInfo [id=follower508, address=akka://test/user/follower508, votingState=VOTING], PeerInfo [id=follower507, address=akka://test/user/follower507, votingState=VOTING], PeerInfo [id=leader506, address=akka://test/user/leader506, votingState=VOTING]] 11:14:20,524 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower509 in 152.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:20,525 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: Recovery completed in 1.371 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:20.622] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader506]: org.opendaylight.controller.cluster.common.actor.Monitor@4de4d331 11:14:20,622 AM [test-akka.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader506] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][02/27/2024 11:14:20.622] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader506] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:20,623 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader506 with journal batch size 1000 11:14:20,624 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: SnapshotOffer called. 11:14:20,624 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Moving last applied index from -1 to -1 11:14:20,624 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Set currentTerm=1, votedFor=null 11:14:20,624 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Updated server config: isVoting: true, peers: [PeerInfo [id=follower508, address=akka://test/user/follower508, votingState=VOTING], PeerInfo [id=follower507, address=akka://test/user/follower507, votingState=VOTING], PeerInfo [id=follower509, address=akka://test/user/follower509, votingState=NON_VOTING]] 11:14:20,624 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader506 in 177.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:20,625 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Recovery completed in 1.209 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:21,45 AM [test-akka.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:14:21,45 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.238:25520, Removed) 11:14:21,45 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Follower): Received ElectionTimeout - switching to Candidate 11:14:21,46 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): Election: Candidate has following voting peers: [follower508, follower507] 11:14:21,46 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Set currentTerm=2, votedFor=leader506 11:14:21,46 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): Starting new election term 2 11:14:21,46 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): Sending RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] to peer follower508 11:14:21,46 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): Sending RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] to peer follower507 11:14:21,46 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:21,46 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:14:21,46 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): Cluster state: [] 11:14:21,47 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:21,47 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:21,47 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Set currentTerm=2, votedFor=null 11:14:21,47 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:21,47 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): In requestVote: RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:21,47 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: Set currentTerm=2, votedFor=leader506 11:14:21,47 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): Cluster state: [] 11:14:21,47 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:21,47 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:21,47 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Set currentTerm=2, votedFor=null 11:14:21,47 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:21,47 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): In requestVote: RequestVote [term=2, candidateId=leader506, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:21,47 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: Set currentTerm=2, votedFor=leader506 11:14:21,47 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:21,49 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:14:21,49 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:14:21,49 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Election: Leader has following peers: [follower508, follower507, follower509] 11:14:21,49 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower508: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:21,49 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower507: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:21,49 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower509: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:21,49 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:14:21,49 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower508: Last sync leader does not match current leader leader506, need to catch up to -1 11:14:21,49 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower507: Last sync leader does not match current leader leader506, need to catch up to -1 11:14:21,50 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower509 (Follower): Term 2 in "AppendEntries [leaderId=leader506, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:14:21,50 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: Set currentTerm=2, votedFor=null 11:14:21,50 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower508 updated: matchIndex: -1, nextIndex: 0 11:14:21,50 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower509: Last sync leader does not match current leader leader506, need to catch up to -1 11:14:21,50 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower508: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:21.050] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower508] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1c63a0e3 11:14:21,50 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower507 updated: matchIndex: -1, nextIndex: 0 11:14:21,50 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower507: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:21.050] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower507] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@59613d0b 11:14:21,50 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower509 updated: matchIndex: -1, nextIndex: 0 11:14:21,50 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower509: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [INFO] [akkaDeadLetter][02/27/2024 11:14:21.050] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower508] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower508] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [02/27/2024 11:14:21.051] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower509] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5c3613f 11:14:21,51 AM [test-akka.actor.default-dispatcher-16] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower508] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:21,51 AM [test-akka.actor.default-dispatcher-16] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower507] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:21,51 AM [test-akka.actor.default-dispatcher-16] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower509] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][02/27/2024 11:14:21.051] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower507] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower507] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][02/27/2024 11:14:21.051] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower509] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower509] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:21,80 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:14:21,80 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader506 starting 11:14:21,83 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: onShutDown 11:14:21,83 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: Initiating leader transfer 11:14:21,84 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower507: onLeaderTransitioning: LeaderTransitioning [leaderId=leader506] 11:14:21,84 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower508: onLeaderTransitioning: LeaderTransitioning [leaderId=leader506] 11:14:21,84 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower509: onLeaderTransitioning: LeaderTransitioning [leaderId=leader506] 11:14:21,84 AM [test-akka.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader506: pauseLeader successfully completed - doing transfer 11:14:21,84 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Attempting to transfer leadership 11:14:21,84 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower508: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:21,84 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower507: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:21,84 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Checking sendAppendEntries for follower follower509: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:21,84 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower507: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 11:14:21,85 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower509: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 11:14:21,85 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower508: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) [WARN] [02/27/2024 11:14:21.085] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower507] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@46da91d8 [WARN] [02/27/2024 11:14:21.085] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower509] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@e9a9c3b [WARN] [02/27/2024 11:14:21.085] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower508] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1d058c1d 11:14:21,85 AM [test-akka.actor.default-dispatcher-5] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower507] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][02/27/2024 11:14:21.085] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower507] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower507] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][02/27/2024 11:14:21.085] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower509] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower509] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:21,85 AM [test-akka.actor.default-dispatcher-5] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower509] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:21,86 AM [test-akka.actor.default-dispatcher-15] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower508] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][02/27/2024 11:14:21.086] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower508] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower508] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 11:14:21,166 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower507: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 11:14:21,166 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower508: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 11:14:21,166 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower509: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 11:14:21,285 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower508: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 11:14:21,285 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower507: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 11:14:21,286 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower509: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 11:14:21,406 AM [test-akka.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader506: leader transfer aborted 11:14:21,406 AM [test-akka.actor.default-dispatcher-16] [WARN] RaftActorLeadershipTransferCohort - leader506: Failed to transfer leadership in 322.3 ms 11:14:21,406 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506: leader transfer failed - sending PoisonPill 11:14:21,406 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader506 (Leader): Leadership transfer expired 11:14:21,407 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower507: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 11:14:21,407 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower508: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) 11:14:21,407 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower509: Lagging 0 entries behind leader leader506 and reached -1 (of expected -1) [WARN] [02/27/2024 11:14:21.407] [test-akka.actor.default-dispatcher-15] [akka://test/user/leader506] received dead letter from TestActor[akka://test/user/follower507]: AppendEntriesReply [term=2, success=true, followerId=follower507, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [02/27/2024 11:14:21.407] [test-akka.actor.default-dispatcher-15] [akka://test/user/leader506] received dead letter from TestActor[akka://test/user/follower508]: AppendEntriesReply [term=2, success=true, followerId=follower508, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [02/27/2024 11:14:21.408] [test-akka.actor.default-dispatcher-15] [akka://test/user/leader506] received dead letter from TestActor[akka://test/user/follower509]: AppendEntriesReply [term=2, success=true, followerId=follower509, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:14:21,409 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader506 ending 11:14:21,411 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 11:14:21,411 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower507-notifier510#-285229199] 11:14:21,423 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower507-collector511#350953320] 11:14:21,423 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower507] 11:14:21,424 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower508-notifier512#-1064417349] 11:14:21,424 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower508-collector513#2021241469] 11:14:21,424 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower508] 11:14:21,425 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower509-notifier514#220453927] 11:14:21,425 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower509-collector515#-1211226856] 11:14:21,425 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower509] 11:14:21,425 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader506-notifier516#-267491085] 11:14:21,426 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader506-collector517#-1010304564] 11:14:21,426 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader506] 11:14:21,427 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 11:14:21,428 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [02/27/2024 11:14:21.433] [test-akka.actor.default-dispatcher-16] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower519]: org.opendaylight.controller.cluster.common.actor.Monitor@53f4e5d9 11:14:21,434 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower519 with journal batch size 1000 11:14:21,435 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519: SnapshotOffer called. 11:14:21,435 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519: Moving last applied index from -1 to -1 11:14:21,435 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519: Set currentTerm=1, votedFor=null 11:14:21,435 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519: Updated server config: isVoting: true, peers: [PeerInfo [id=follower520, address=akka://test/user/follower520, votingState=VOTING], PeerInfo [id=leader518, address=akka://test/user/leader518, votingState=VOTING], PeerInfo [id=follower521, address=akka://test/user/follower521, votingState=NON_VOTING]] 11:14:21,436 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower519 in 132.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:21,438 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower519: Recovery completed in 2.407 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:21.438] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower520]: org.opendaylight.controller.cluster.common.actor.Monitor@5a8f76b4 11:14:21,440 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower520 with journal batch size 1000 11:14:21,441 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520: SnapshotOffer called. 11:14:21,441 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520: Moving last applied index from -1 to -1 11:14:21,441 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520: Set currentTerm=1, votedFor=null 11:14:21,441 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520: Updated server config: isVoting: true, peers: [PeerInfo [id=follower519, address=akka://test/user/follower519, votingState=VOTING], PeerInfo [id=leader518, address=akka://test/user/leader518, votingState=VOTING], PeerInfo [id=follower521, address=akka://test/user/follower521, votingState=NON_VOTING]] 11:14:21,442 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower520 in 108.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:21,443 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower520: Recovery completed in 1.271 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:21.541] [test-akka.actor.default-dispatcher-15] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower521]: org.opendaylight.controller.cluster.common.actor.Monitor@541e1a60 11:14:21,542 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower521 with journal batch size 1000 11:14:21,543 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower521: SnapshotOffer called. 11:14:21,543 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower521: Moving last applied index from -1 to -1 11:14:21,543 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower521: Set currentTerm=1, votedFor=null 11:14:21,543 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower521: Updated server config: isVoting: false, peers: [PeerInfo [id=follower519, address=akka://test/user/follower519, votingState=VOTING], PeerInfo [id=follower520, address=akka://test/user/follower520, votingState=VOTING], PeerInfo [id=leader518, address=akka://test/user/leader518, votingState=VOTING]] 11:14:21,543 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower521 in 90.62 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [02/27/2024 11:14:21.544] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader518]: org.opendaylight.controller.cluster.common.actor.Monitor@edb8cb4 11:14:21,545 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader518 with journal batch size 1000 11:14:21,545 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower521: Recovery completed in 2.281 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:21,546 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: SnapshotOffer called. 11:14:21,546 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: Moving last applied index from -1 to -1 11:14:21,546 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: Set currentTerm=1, votedFor=null 11:14:21,546 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: Updated server config: isVoting: true, peers: [PeerInfo [id=follower519, address=akka://test/user/follower519, votingState=VOTING], PeerInfo [id=follower520, address=akka://test/user/follower520, votingState=VOTING], PeerInfo [id=follower521, address=akka://test/user/follower521, votingState=NON_VOTING]] 11:14:21,546 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader518 in 126.0 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:21,547 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: Recovery completed in 1.007 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:21,965 AM [test-akka.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:14:21,965 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.238:25520, Removed) 11:14:21,965 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Follower): Received ElectionTimeout - switching to Candidate 11:14:21,965 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Candidate): Election: Candidate has following voting peers: [follower519, follower520] 11:14:21,965 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: Set currentTerm=2, votedFor=leader518 11:14:21,966 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Candidate): Starting new election term 2 11:14:21,966 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Candidate): Sending RequestVote [term=2, candidateId=leader518, lastLogIndex=-1, lastLogTerm=-1] to peer follower519 11:14:21,966 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Candidate): Sending RequestVote [term=2, candidateId=leader518, lastLogIndex=-1, lastLogTerm=-1] to peer follower520 11:14:21,966 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:21,966 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:14:21,966 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:21,966 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): Cluster state: [] 11:14:21,966 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader518, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:21,966 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Follower): Cluster state: [] 11:14:21,966 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader518, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:21,966 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader518, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:21,966 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519: Set currentTerm=2, votedFor=null 11:14:21,966 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader518, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:21,966 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520: Set currentTerm=2, votedFor=null 11:14:21,967 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Follower): In requestVote: RequestVote [term=2, candidateId=leader518, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:21,967 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): In requestVote: RequestVote [term=2, candidateId=leader518, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:21,967 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520: Set currentTerm=2, votedFor=leader518 11:14:21,967 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519: Set currentTerm=2, votedFor=leader518 11:14:21,967 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:21,967 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:21,967 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:14:21,967 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:14:21,968 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Election: Leader has following peers: [follower519, follower520, follower521] 11:14:21,968 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower519: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:21,968 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower520: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:21,968 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower521: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:21,968 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:14:21,968 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower521 (Follower): Term 2 in "AppendEntries [leaderId=leader518, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:14:21,968 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower521: Set currentTerm=2, votedFor=null 11:14:21,968 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower519: Last sync leader does not match current leader leader518, need to catch up to -1 11:14:21,968 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower520: Last sync leader does not match current leader leader518, need to catch up to -1 11:14:21,968 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower521: Last sync leader does not match current leader leader518, need to catch up to -1 11:14:21,969 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower519 updated: matchIndex: -1, nextIndex: 0 11:14:21,969 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower519: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:21.969] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower519] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@65aea9e6 11:14:21,969 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower520 updated: matchIndex: -1, nextIndex: 0 11:14:21,969 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower520: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:21,969 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower521 updated: matchIndex: -1, nextIndex: 0 11:14:21,969 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower521: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:21.969] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower520] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@15160787 [WARN] [02/27/2024 11:14:21.975] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower521] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@477a97bd 11:14:21,997 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:14:21,997 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:14:21,998 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:14:21,998 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:14:21,999 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:14:21,999 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: onRequestLeadership RequestLeadership [requestedFollowerId=follower520, replyTo=Actor[akka://test/user/$a#-44218330]] 11:14:22,1 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: Initiating leader transfer 11:14:22,1 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader518: pauseLeader successfully completed - doing transfer 11:14:22,1 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Attempting to transfer leadership 11:14:22,1 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower519: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,1 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower520: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,1 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower521: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,2 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower519: Lagging 0 entries behind leader leader518 and reached -1 (of expected -1) 11:14:22,2 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519: onLeaderTransitioning: LeaderTransitioning [leaderId=leader518] 11:14:22,2 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower521: Lagging 0 entries behind leader leader518 and reached -1 (of expected -1) 11:14:22,2 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower521: onLeaderTransitioning: LeaderTransitioning [leaderId=leader518] [WARN] [02/27/2024 11:14:22.002] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower519] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@37258a8c [WARN] [02/27/2024 11:14:22.002] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower521] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@441d6c32 11:14:22,3 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower520: Lagging 0 entries behind leader leader518 and reached -1 (of expected -1) 11:14:22,3 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520: onLeaderTransitioning: LeaderTransitioning [leaderId=leader518] 11:14:22,3 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): tryToCompleteLeadershipTransfer: followerId: follower520, matchIndex: -1, lastIndex: -1, isVoting: true 11:14:22,3 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Follower's log matches - sending ElectionTimeout 11:14:22,3 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower519: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,3 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower520: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,3 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Checking sendAppendEntries for follower follower521: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,3 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Leader transfer complete 11:14:22,3 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader518: leader transfer complete - waiting for new leader [WARN] [02/27/2024 11:14:22.004] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower520] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@69173308 11:14:22,4 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower519: Lagging 0 entries behind leader leader518 and reached -1 (of expected -1) 11:14:22,4 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower520: Lagging 0 entries behind leader leader518 and reached -1 (of expected -1) 11:14:22,4 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower521: Lagging 0 entries behind leader leader518 and reached -1 (of expected -1) 11:14:22,4 AM [test-akka.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:14:22,4 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Follower): Received TimeoutNow - switching to Candidate 11:14:22,4 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Candidate): Election: Candidate has following voting peers: [follower519, leader518] 11:14:22,4 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520: Set currentTerm=3, votedFor=follower520 11:14:22,4 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Candidate): Starting new election term 3 11:14:22,4 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Candidate): Sending RequestVote [term=3, candidateId=follower520, lastLogIndex=-1, lastLogTerm=-1] to peer follower519 11:14:22,4 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Candidate): Sending RequestVote [term=3, candidateId=follower520, lastLogIndex=-1, lastLogTerm=-1] to peer leader518 11:14:22,5 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 11:14:22,5 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): Cluster state: [] 11:14:22,5 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower520, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:22,5 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower520, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 2 - updating term 11:14:22,5 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519: Set currentTerm=3, votedFor=null 11:14:22,5 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): In requestVote: RequestVote [term=3, candidateId=follower520, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:22,5 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519: Set currentTerm=3, votedFor=follower520 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Cluster state: [] 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower520, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:22,5 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower519 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower520, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: Set currentTerm=3, votedFor=null 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): Leadership transfer in progress - processing RequestVote 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): In requestVote: RequestVote [term=3, candidateId=follower520, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: Set currentTerm=3, votedFor=follower520 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 11:14:22,5 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader518 (Leader) :- Switching from behavior Leader to Follower, election term: 3 11:14:22,9 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 11:14:22,9 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 11:14:22,9 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Leader): Election: Leader has following peers: [follower519, leader518, follower521] 11:14:22,9 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Leader): Checking sendAppendEntries for follower follower519: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,10 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Leader): Checking sendAppendEntries for follower leader518: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,10 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Leader): Checking sendAppendEntries for follower follower521: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,10 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 11:14:22,10 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower519: Last sync leader does not match current leader follower520, need to catch up to -1 11:14:22,10 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower519 updated: matchIndex: -1, nextIndex: 0 11:14:22,10 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Leader): Checking sendAppendEntries for follower follower519: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:22.010] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower519] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5f7f7272 11:14:22,10 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower521 (Follower): Term 3 in "AppendEntries [leaderId=follower520, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 11:14:22,10 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower521: Set currentTerm=3, votedFor=null 11:14:22,10 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - leader518: Last sync leader does not match current leader follower520, need to catch up to -1 11:14:22,10 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower521: Last sync leader does not match current leader follower520, need to catch up to -1 11:14:22,10 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader518: leader changed to follower520 11:14:22,11 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader518 updated: matchIndex: -1, nextIndex: 0 11:14:22,11 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Leader): Checking sendAppendEntries for follower leader518: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,11 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower521 updated: matchIndex: -1, nextIndex: 0 11:14:22,11 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower520 (Leader): Checking sendAppendEntries for follower follower521: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,11 AM [test-akka.actor.default-dispatcher-17] [INFO] RaftActorLeadershipTransferCohort - leader518: Successfully transferred leadership to follower520 in 9.669 ms 11:14:22,11 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader518: Leadership transferred successfully to follower520 [WARN] [02/27/2024 11:14:22.011] [test-akka.actor.default-dispatcher-15] [akka://test/user/leader518] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6db911a0 [WARN] [02/27/2024 11:14:22.012] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower521] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@42074ac5 11:14:22,54 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 11:14:22,54 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower519-notifier522#-965403141] 11:14:22,66 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower519-collector523#518825835] 11:14:22,66 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower519] 11:14:22,66 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower520-notifier524#-1664764089] 11:14:22,66 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower520-collector525#-238682918] 11:14:22,67 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower520] 11:14:22,67 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower521-notifier526#1969580416] 11:14:22,67 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower521-collector527#505384646] 11:14:22,67 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower521] 11:14:22,67 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader518-notifier528#-136494661] 11:14:22,68 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader518-collector529#452360495] 11:14:22,68 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader518] 11:14:22,68 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$a#-44218330] 11:14:22,69 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 11:14:22,69 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [02/27/2024 11:14:22.072] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower531]: org.opendaylight.controller.cluster.common.actor.Monitor@72133f5f 11:14:22,73 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower531 with journal batch size 1000 11:14:22,74 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531: SnapshotOffer called. 11:14:22,74 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531: Moving last applied index from -1 to -1 11:14:22,74 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531: Set currentTerm=1, votedFor=null 11:14:22,74 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531: Updated server config: isVoting: true, peers: [PeerInfo [id=leader530, address=akka://test/user/leader530, votingState=VOTING], PeerInfo [id=follower533, address=akka://test/user/follower533, votingState=NON_VOTING], PeerInfo [id=follower532, address=akka://test/user/follower532, votingState=VOTING]] 11:14:22,74 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower531 in 113.8 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:22,75 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower531: Recovery completed in 1.189 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:22.177] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower532]: org.opendaylight.controller.cluster.common.actor.Monitor@520a9962 11:14:22,178 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower532 with journal batch size 1000 11:14:22,178 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532: SnapshotOffer called. 11:14:22,178 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532: Moving last applied index from -1 to -1 11:14:22,178 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532: Set currentTerm=1, votedFor=null 11:14:22,179 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532: Updated server config: isVoting: true, peers: [PeerInfo [id=leader530, address=akka://test/user/leader530, votingState=VOTING], PeerInfo [id=follower531, address=akka://test/user/follower531, votingState=VOTING], PeerInfo [id=follower533, address=akka://test/user/follower533, votingState=NON_VOTING]] 11:14:22,179 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower532 in 130.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:22,180 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower532: Recovery completed in 1.531 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:22.181] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower533]: org.opendaylight.controller.cluster.common.actor.Monitor@110f77b8 11:14:22,182 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower533 with journal batch size 1000 11:14:22,187 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower533: SnapshotOffer called. 11:14:22,187 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower533: Moving last applied index from -1 to -1 11:14:22,187 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower533: Set currentTerm=1, votedFor=null 11:14:22,187 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower533: Updated server config: isVoting: false, peers: [PeerInfo [id=leader530, address=akka://test/user/leader530, votingState=VOTING], PeerInfo [id=follower531, address=akka://test/user/follower531, votingState=VOTING], PeerInfo [id=follower532, address=akka://test/user/follower532, votingState=VOTING]] 11:14:22,188 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower533 in 90.05 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:22,188 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower533: Recovery completed in 974.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:22,285 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader530 with journal batch size 1000 [WARN] [02/27/2024 11:14:22.285] [test-akka.actor.default-dispatcher-15] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader530]: org.opendaylight.controller.cluster.common.actor.Monitor@4b186813 11:14:22,285 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530: SnapshotOffer called. 11:14:22,286 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530: Moving last applied index from -1 to -1 11:14:22,286 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530: Set currentTerm=1, votedFor=null 11:14:22,286 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530: Updated server config: isVoting: true, peers: [PeerInfo [id=follower531, address=akka://test/user/follower531, votingState=VOTING], PeerInfo [id=follower533, address=akka://test/user/follower533, votingState=NON_VOTING], PeerInfo [id=follower532, address=akka://test/user/follower532, votingState=VOTING]] 11:14:22,286 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader530 in 183.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:22,288 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader530: Recovery completed in 2.093 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:22,635 AM [test-akka.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:14:22,636 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.238:25520, Removed) 11:14:22,636 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Follower): Received ElectionTimeout - switching to Candidate 11:14:22,636 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Candidate): Election: Candidate has following voting peers: [follower531, follower532] 11:14:22,636 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530: Set currentTerm=2, votedFor=leader530 11:14:22,636 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Candidate): Starting new election term 2 11:14:22,637 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Candidate): Sending RequestVote [term=2, candidateId=leader530, lastLogIndex=-1, lastLogTerm=-1] to peer follower531 11:14:22,637 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Candidate): Sending RequestVote [term=2, candidateId=leader530, lastLogIndex=-1, lastLogTerm=-1] to peer follower532 11:14:22,637 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:14:22,637 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:22,637 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:22,637 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531 (Follower): Cluster state: [] 11:14:22,637 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532 (Follower): Cluster state: [] 11:14:22,637 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader530, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:22,637 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader530, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:22,637 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower531 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader530, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:22,637 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower532 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader530, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:22,637 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531: Set currentTerm=2, votedFor=null 11:14:22,637 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532: Set currentTerm=2, votedFor=null 11:14:22,637 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532 (Follower): In requestVote: RequestVote [term=2, candidateId=leader530, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:22,637 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532: Set currentTerm=2, votedFor=leader530 11:14:22,637 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531 (Follower): In requestVote: RequestVote [term=2, candidateId=leader530, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:22,637 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531: Set currentTerm=2, votedFor=leader530 11:14:22,638 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:22,638 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:22,639 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:14:22,639 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:14:22,639 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Election: Leader has following peers: [follower531, follower533, follower532] 11:14:22,639 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Checking sendAppendEntries for follower follower531: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,639 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Checking sendAppendEntries for follower follower533: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,639 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Checking sendAppendEntries for follower follower532: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,639 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower531: Last sync leader does not match current leader leader530, need to catch up to -1 11:14:22,639 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:14:22,639 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower533 (Follower): Term 2 in "AppendEntries [leaderId=leader530, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:14:22,639 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower533: Set currentTerm=2, votedFor=null 11:14:22,640 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower532: Last sync leader does not match current leader leader530, need to catch up to -1 11:14:22,640 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower531 updated: matchIndex: -1, nextIndex: 0 11:14:22,640 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower533: Last sync leader does not match current leader leader530, need to catch up to -1 11:14:22,640 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Checking sendAppendEntries for follower follower531: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:22.640] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower531] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@18551227 11:14:22,640 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower532 updated: matchIndex: -1, nextIndex: 0 [WARN] [02/27/2024 11:14:22.640] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower532] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3884bc41 11:14:22,640 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Checking sendAppendEntries for follower follower532: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,640 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower533 updated: matchIndex: -1, nextIndex: 0 11:14:22,640 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Checking sendAppendEntries for follower follower533: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:22.641] [test-akka.actor.default-dispatcher-7] [akka://test/user/follower533] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7b654b05 11:14:22,642 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:14:22,642 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:14:22,643 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:14:22,643 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower532 starting 11:14:22,643 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower532: onShutDown 11:14:22,644 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower532 ending 11:14:22,644 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:14:22,644 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:14:22,644 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530: onRequestLeadership RequestLeadership [requestedFollowerId=follower532, replyTo=Actor[akka://test/user/$b#-1590971281]] 11:14:22,644 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530: Initiating leader transfer 11:14:22,644 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader530: pauseLeader successfully completed - doing transfer 11:14:22,644 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Attempting to transfer leadership 11:14:22,644 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Checking sendAppendEntries for follower follower531: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,644 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower531: onLeaderTransitioning: LeaderTransitioning [leaderId=leader530] 11:14:22,644 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower533: onLeaderTransitioning: LeaderTransitioning [leaderId=leader530] 11:14:22,644 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Checking sendAppendEntries for follower follower533: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:22,644 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Checking sendAppendEntries for follower follower532: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:22.644] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower532] received dead letter from TestActor[akka://test/user/leader530]: LeaderTransitioning [leaderId=leader530] 11:14:22,645 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower533: Lagging 0 entries behind leader leader530 and reached -1 (of expected -1) 11:14:22,645 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower531: Lagging 0 entries behind leader leader530 and reached -1 (of expected -1) [WARN] [02/27/2024 11:14:22.645] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower532] received dead letter from TestActor[akka://test/user/leader530]: AppendEntries [leaderId=leader530, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [02/27/2024 11:14:22.645] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower533] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7ad9d499 [WARN] [02/27/2024 11:14:22.645] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower531] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5ac2eca3 11:14:22,756 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower531: Lagging 0 entries behind leader leader530 and reached -1 (of expected -1) 11:14:22,756 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower533: Lagging 0 entries behind leader leader530 and reached -1 (of expected -1) [WARN] [02/27/2024 11:14:22.756] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower532] received dead letter from TestActor[akka://test/user/leader530]: AppendEntries [leaderId=leader530, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:14:22,875 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower531: Lagging 0 entries behind leader leader530 and reached -1 (of expected -1) 11:14:22,876 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower533: Lagging 0 entries behind leader leader530 and reached -1 (of expected -1) [WARN] [02/27/2024 11:14:22.876] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower532] received dead letter from TestActor[akka://test/user/leader530]: AppendEntries [leaderId=leader530, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:14:22,996 AM [test-akka.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader530: leader transfer aborted 11:14:22,996 AM [test-akka.actor.default-dispatcher-15] [WARN] RaftActorLeadershipTransferCohort - leader530: Failed to transfer leadership in 351.8 ms 11:14:22,996 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530: LeadershipTransfer request from follower532 failed 11:14:22,998 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader530 (Leader): Leadership transfer expired 11:14:22,998 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower531: Lagging 0 entries behind leader leader530 and reached -1 (of expected -1) 11:14:22,998 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower533: Lagging 0 entries behind leader leader530 and reached -1 (of expected -1) [WARN] [02/27/2024 11:14:22.998] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower532] received dead letter from TestActor[akka://test/user/leader530]: AppendEntries [leaderId=leader530, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:14:22,999 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 11:14:22,999 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower531-notifier534#-1070955175] 11:14:23,10 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower531-collector535#-2054080303] 11:14:23,10 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower531] 11:14:23,11 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower532-notifier536#-777783216] 11:14:23,11 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower532-collector537#1216330726] 11:14:23,11 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower532] 11:14:23,11 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower533-notifier538#1899421337] 11:14:23,12 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower533-collector539#-891616539] 11:14:23,12 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower533] 11:14:23,12 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader530-notifier540#1688941729] 11:14:23,13 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader530-collector541#-2107272832] 11:14:23,13 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader530] 11:14:23,13 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$b#-1590971281] 11:14:23,15 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 11:14:23,15 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:14:23,18 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower543 with journal batch size 1000 [WARN] [02/27/2024 11:14:23.019] [test-akka.actor.default-dispatcher-7] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower543]: org.opendaylight.controller.cluster.common.actor.Monitor@72cf6829 11:14:23,20 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: SnapshotOffer called. 11:14:23,20 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Moving last applied index from -1 to -1 11:14:23,20 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Set currentTerm=1, votedFor=null 11:14:23,20 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Updated server config: isVoting: true, peers: [PeerInfo [id=leader542, address=akka://test/user/leader542, votingState=VOTING], PeerInfo [id=follower545, address=akka://test/user/follower545, votingState=NON_VOTING], PeerInfo [id=follower544, address=akka://test/user/follower544, votingState=VOTING]] 11:14:23,20 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower543 in 171.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:23,22 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Recovery completed in 1.938 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:23.121] [test-akka.actor.default-dispatcher-7] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower544]: org.opendaylight.controller.cluster.common.actor.Monitor@7c39243c 11:14:23,122 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower544 with journal batch size 1000 11:14:23,123 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: SnapshotOffer called. 11:14:23,123 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Moving last applied index from -1 to -1 11:14:23,123 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Set currentTerm=1, votedFor=null 11:14:23,123 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Updated server config: isVoting: true, peers: [PeerInfo [id=leader542, address=akka://test/user/leader542, votingState=VOTING], PeerInfo [id=follower545, address=akka://test/user/follower545, votingState=NON_VOTING], PeerInfo [id=follower543, address=akka://test/user/follower543, votingState=VOTING]] 11:14:23,124 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower544 in 154.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:23,126 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Recovery completed in 3.049 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:23.224] [test-akka.actor.default-dispatcher-16] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower545]: org.opendaylight.controller.cluster.common.actor.Monitor@39925562 11:14:23,225 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower545 with journal batch size 1000 11:14:23,226 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: SnapshotOffer called. 11:14:23,226 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: Moving last applied index from -1 to -1 11:14:23,226 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: Set currentTerm=1, votedFor=null 11:14:23,226 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: Updated server config: isVoting: false, peers: [PeerInfo [id=leader542, address=akka://test/user/leader542, votingState=VOTING], PeerInfo [id=follower544, address=akka://test/user/follower544, votingState=VOTING], PeerInfo [id=follower543, address=akka://test/user/follower543, votingState=VOTING]] 11:14:23,226 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower545 in 162.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:23,228 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: Recovery completed in 2.130 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:23.328] [test-akka.actor.default-dispatcher-16] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader542]: org.opendaylight.controller.cluster.common.actor.Monitor@6f2ca6c9 11:14:23,329 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader542 with journal batch size 1000 11:14:23,330 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: SnapshotOffer called. 11:14:23,330 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Moving last applied index from -1 to -1 11:14:23,330 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Set currentTerm=1, votedFor=null 11:14:23,330 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Updated server config: isVoting: true, peers: [PeerInfo [id=follower545, address=akka://test/user/follower545, votingState=NON_VOTING], PeerInfo [id=follower544, address=akka://test/user/follower544, votingState=VOTING], PeerInfo [id=follower543, address=akka://test/user/follower543, votingState=VOTING]] 11:14:23,331 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader542 in 107.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:23,332 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Recovery completed in 1.521 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:23,665 AM [test-akka.actor.default-dispatcher-7] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:14:23,665 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.238:25520, Removed) 11:14:23,665 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Follower): Received ElectionTimeout - switching to Candidate 11:14:23,666 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Candidate): Election: Candidate has following voting peers: [follower544, follower543] 11:14:23,666 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Set currentTerm=2, votedFor=leader542 11:14:23,666 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Candidate): Starting new election term 2 11:14:23,666 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Candidate): Sending RequestVote [term=2, candidateId=leader542, lastLogIndex=-1, lastLogTerm=-1] to peer follower544 11:14:23,666 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Candidate): Sending RequestVote [term=2, candidateId=leader542, lastLogIndex=-1, lastLogTerm=-1] to peer follower543 11:14:23,666 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:14:23,666 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:23,666 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Cluster state: [] 11:14:23,666 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader542, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:23,666 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader542, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:23,666 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Set currentTerm=2, votedFor=null 11:14:23,666 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:23,666 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): Cluster state: [] 11:14:23,666 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): In requestVote: RequestVote [term=2, candidateId=leader542, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:23,666 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Set currentTerm=2, votedFor=leader542 11:14:23,666 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader542, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:23,667 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader542, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:23,667 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Set currentTerm=2, votedFor=null 11:14:23,667 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:23,667 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): In requestVote: RequestVote [term=2, candidateId=leader542, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:23,667 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Set currentTerm=2, votedFor=leader542 11:14:23,667 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:23,667 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:14:23,668 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:14:23,668 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Election: Leader has following peers: [follower545, follower544, follower543] 11:14:23,668 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower545: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:23,668 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower544: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:23,668 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): Term 2 in "AppendEntries [leaderId=leader542, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:14:23,668 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower543: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:23,668 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: Set currentTerm=2, votedFor=null 11:14:23,668 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:14:23,668 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower545: Last sync leader does not match current leader leader542, need to catch up to -1 11:14:23,669 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower543: Last sync leader does not match current leader leader542, need to catch up to -1 11:14:23,669 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower544: Last sync leader does not match current leader leader542, need to catch up to -1 11:14:23,669 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower545 updated: matchIndex: -1, nextIndex: 0 11:14:23,669 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower545: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:23.669] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower543] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@92714ce [WARN] [02/27/2024 11:14:23.669] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower544] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5e99cc69 11:14:23,669 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower543 updated: matchIndex: -1, nextIndex: 0 11:14:23,669 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower543: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:23,669 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower544 updated: matchIndex: -1, nextIndex: 0 11:14:23,669 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower544: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:23.670] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower545] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6dc8e585 11:14:23,680 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:14:23,680 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 11:14:23,680 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:23,680 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:23,680 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, payload: class org.opendaylight.controller.cluster.raft.MockRaftActorContext$MockPayload, isSendImmediate: true 11:14:23,680 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower545: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,681 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): sendAppendEntries: 0 is present for follower follower545 11:14:23,681 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Sending AppendEntries to follower follower545: AppendEntries [leaderId=leader542, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:23,681 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower544: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,681 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): sendAppendEntries: 0 is present for follower follower544 11:14:23,681 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Sending AppendEntries to follower follower544: AppendEntries [leaderId=leader542, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:23,681 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): handleAppendEntries: AppendEntries [leaderId=leader542, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:23,681 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower543: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,681 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): sendAppendEntries: 0 is present for follower follower543 11:14:23,681 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): Number of entries to be appended = 1 11:14:23,681 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:14:23,681 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): Append entry to log zero 11:14:23,681 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:23,681 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Sending AppendEntries to follower follower543: AppendEntries [leaderId=leader542, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:23,681 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): Log size is now 1 11:14:23,681 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): handleAppendEntries: AppendEntries [leaderId=leader542, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:23,681 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower545, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:14:23,681 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): Number of entries to be appended = 1 11:14:23,681 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower545: Lagging 0 entries behind leader leader542 and reached -1 (of expected -1) 11:14:23,681 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:14:23,681 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): Append entry to log zero 11:14:23,681 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:23,681 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower545 updated: matchIndex: 0, nextIndex: 1 11:14:23,681 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower545: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,681 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): Log size is now 1 11:14:23,681 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower543, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:14:23,682 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower543: Lagging 0 entries behind leader leader542 and reached -1 (of expected -1) 11:14:23,682 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower543 updated: matchIndex: 0, nextIndex: 1 11:14:23,682 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower543: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,683 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:23,683 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 11:14:23,683 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:23,683 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Setting last applied to 0 11:14:23,683 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Moving last applied index from -1 to 0 11:14:23,683 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Applying state for log index 0 data zero 11:14:23,683 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [02/27/2024 11:14:23.683] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower545] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@344b1c0f [WARN] [02/27/2024 11:14:23.683] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower543] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2b311ff0 11:14:23,684 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: applyState called: zero 11:14:23,684 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Persisting ApplyJournalEntries with index=0 11:14:23,786 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): sendAppendEntries: 0 is present for follower follower544 11:14:23,786 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): Commit index set to 0 11:14:23,786 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Sending AppendEntries to follower follower544: AppendEntries [leaderId=leader542, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:23,786 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower545: Lagging 0 entries behind leader leader542 and reached 0 (of expected -1) 11:14:23,786 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:14:23,786 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): Setting last applied to 0 11:14:23,786 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: Moving last applied index from -1 to 0 11:14:23,786 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: Applying state for log index 0 data zero 11:14:23,786 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): Commit index set to 0 11:14:23,786 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: applyState called: zero 11:14:23,786 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower543: Lagging 0 entries behind leader leader542 and reached 0 (of expected -1) 11:14:23,786 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:14:23,786 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): Setting last applied to 0 11:14:23,786 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Moving last applied index from -1 to 0 11:14:23,786 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Applying state for log index 0 data zero 11:14:23,786 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: Persisting ApplyJournalEntries with index=0 11:14:23,786 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: applyState called: zero 11:14:23,787 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Persisting ApplyJournalEntries with index=0 11:14:23,831 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 11:14:23,831 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 11:14:23,832 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: onShutDown 11:14:23,832 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Initiating leader transfer 11:14:23,832 AM [test-akka.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader542: pauseLeader successfully completed - doing transfer 11:14:23,832 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Attempting to transfer leadership 11:14:23,832 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower545: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,832 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: onLeaderTransitioning: LeaderTransitioning [leaderId=leader542] 11:14:23,832 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: onLeaderTransitioning: LeaderTransitioning [leaderId=leader542] 11:14:23,832 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower544: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,832 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): sendAppendEntries: 0 is present for follower follower544 11:14:23,832 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower543: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,832 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower545: Lagging 0 entries behind leader leader542 and reached 0 (of expected -1) 11:14:23,833 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower543: Lagging 0 entries behind leader leader542 and reached 0 (of expected -1) 11:14:23,833 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): tryToCompleteLeadershipTransfer: followerId: follower545, matchIndex: 0, lastIndex: 0, isVoting: false 11:14:23,833 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): tryToCompleteLeadershipTransfer: followerId: follower543, matchIndex: 0, lastIndex: 0, isVoting: true 11:14:23,833 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Follower's log matches - sending ElectionTimeout 11:14:23,833 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower545: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,833 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower544: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,833 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): sendAppendEntries: 0 is present for follower follower544 11:14:23,833 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Checking sendAppendEntries for follower follower543: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,833 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Leader transfer complete 11:14:23,833 AM [test-akka.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader542: leader transfer complete - waiting for new leader 11:14:23,833 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower545: Lagging 0 entries behind leader leader542 and reached 0 (of expected -1) 11:14:23,833 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower543: Lagging 0 entries behind leader leader542 and reached 0 (of expected -1) 11:14:23,833 AM [test-akka.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:14:23,833 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower): Received TimeoutNow - switching to Candidate 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Candidate): Election: Candidate has following voting peers: [leader542, follower544] 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543: Set currentTerm=3, votedFor=follower543 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Candidate): Starting new election term 3 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Candidate): Sending RequestVote [term=3, candidateId=follower543, lastLogIndex=0, lastLogTerm=2] to peer leader542 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Candidate): Sending RequestVote [term=3, candidateId=follower543, lastLogIndex=0, lastLogTerm=2] to peer follower544 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 11:14:23,834 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:23,834 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Cluster state: [] 11:14:23,834 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower543, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:23,834 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower543, lastLogIndex=0, lastLogTerm=2]" message is greater than leader's term 2 - switching to Follower 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Cluster state: [] 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower543, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower543, lastLogIndex=0, lastLogTerm=2]" message is greater than follower's term 2 - updating term 11:14:23,834 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Set currentTerm=3, votedFor=null 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Set currentTerm=3, votedFor=null 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): In requestVote: RequestVote [term=3, candidateId=follower543, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Set currentTerm=3, votedFor=follower543 11:14:23,834 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): Leadership transfer in progress - processing RequestVote 11:14:23,834 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): In requestVote: RequestVote [term=3, candidateId=follower543, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 11:14:23,834 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 11:14:23,834 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: Set currentTerm=3, votedFor=follower543 11:14:23,835 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 11:14:23,835 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader542 (Leader) :- Switching from behavior Leader to Follower, election term: 3 11:14:23,837 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 11:14:23,837 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 11:14:23,837 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): Election: Leader has following peers: [leader542, follower545, follower544] 11:14:23,837 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): Checking sendAppendEntries for follower leader542: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,837 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): Checking sendAppendEntries for follower follower545: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,837 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): Checking sendAppendEntries for follower follower544: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,837 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 11:14:23,837 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - leader542: Last sync leader does not match current leader follower543, need to catch up to -1 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [DEBUG] RaftActorLeadershipTransferCohort - leader542: leader changed to follower543 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [INFO] RaftActorLeadershipTransferCohort - leader542: Successfully transferred leadership to follower543 in 5.678 ms 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader542: leader transfer succeeded - sending PoisonPill 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower545 (Follower): Term 3 in "AppendEntries [leaderId=follower543, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower545: Set currentTerm=3, votedFor=null 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower545: Last sync leader does not match current leader follower543, need to catch up to -1 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader542 updated: matchIndex: 0, nextIndex: 1 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): Checking sendAppendEntries for follower leader542: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower545 updated: matchIndex: 0, nextIndex: 1 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): Checking sendAppendEntries for follower follower545: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:23,838 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - leader542: Lagging 0 entries behind leader follower543 and reached 0 (of expected -1) [WARN] [02/27/2024 11:14:23.839] [test-akka.actor.default-dispatcher-7] [akka://test/user/leader542] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@16b45c30 [WARN] [02/27/2024 11:14:23.839] [test-akka.actor.default-dispatcher-7] [akka://test/user/leader542] received dead letter: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@8103059 11:14:23,840 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower545: Lagging 0 entries behind leader follower543 and reached 0 (of expected -1) [WARN] [02/27/2024 11:14:23.840] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower545] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6f78691b [WARN] [02/27/2024 11:14:23.840] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower545] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@45b65dbc [WARN] [02/27/2024 11:14:24.356] [test-akka.actor.default-dispatcher-15] [akka://test/user/leader542] received dead letter from TestActor[akka://test/user/follower543]: AppendEntries [leaderId=follower543, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:14:24,356 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower545: Lagging 0 entries behind leader follower543 and reached 0 (of expected -1) 11:14:24,356 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower544: Last sync leader does not match current leader follower543, need to catch up to -1 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): sendAppendEntries: 0 is present for follower follower544 [WARN] [02/27/2024 11:14:24.357] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower544] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6c3f7fa7 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): Sending AppendEntries to follower follower544: AppendEntries [leaderId=follower543, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): handleAppendEntries: AppendEntries [leaderId=follower543, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Number of entries to be appended = 1 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Append entry to log zero 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Log size is now 1 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Commit index set to 0 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): handleAppendEntries returning : AppendEntriesReply [term=3, success=true, followerId=follower544, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower544: Lagging 0 entries behind leader follower543 and reached 0 (of expected -1) 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544 (Follower): Setting last applied to 0 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Moving last applied index from -1 to 0 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Applying state for log index 0 data zero 11:14:24,357 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower544 updated: matchIndex: 0, nextIndex: 1 11:14:24,357 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower543 (Leader): Checking sendAppendEntries for follower follower544: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:24,357 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: applyState called: zero 11:14:24,359 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:24,359 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: Persisting ApplyJournalEntries with index=0 [WARN] [02/27/2024 11:14:24.359] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower544] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@46915770 11:14:24,387 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: onLeaderTransitioning: LeaderTransitioning [leaderId=leader542] 11:14:24,388 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 11:14:24,388 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower544 starting 11:14:24,389 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower544: onShutDown 11:14:24,390 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower544 ending 11:14:24,390 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 11:14:24,390 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower543-notifier546#690190621] 11:14:24,401 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower543-collector547#1600773813] 11:14:24,402 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower543] 11:14:24,402 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower544-notifier548#609261953] 11:14:24,402 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower544-collector549#-750649892] 11:14:24,403 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower544] 11:14:24,403 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower545-notifier550#1421471183] 11:14:24,403 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower545-collector551#386303085] 11:14:24,403 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower545] 11:14:24,404 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader542-notifier552#1259753557] 11:14:24,404 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader542-collector553#1993811647] 11:14:24,404 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader542] 11:14:24,405 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [02/27/2024 11:14:24.407] [test-akka.actor.default-dispatcher-15] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader554]: org.opendaylight.controller.cluster.common.actor.Monitor@4023e38a 11:14:24,408 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader554 with journal batch size 1000 11:14:24,411 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader554: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:24,412 AM [test-akka.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:14:24,412 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader554 (Follower): Received TimeoutNow - switching to Candidate 11:14:24,412 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader554 (Candidate): Election: Candidate has following voting peers: [] 11:14:24,412 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader554: Set currentTerm=1, votedFor=leader554 11:14:24,412 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader554 (Candidate): Starting new election term 1 11:14:24,412 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader554 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 11:14:24,413 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader554 (Candidate): Received ElectionTimeout 11:14:24,413 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader554 (Leader): Election: Leader has following peers: [] 11:14:24,413 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader554 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 11:14:24,414 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader554 starting 11:14:24,414 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader554: onShutDown 11:14:24,414 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader554 ending 11:14:24,414 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 11:14:24,414 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader554-collector558#1261998494] 11:14:24,425 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader554] 11:14:24,426 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 11:14:24,426 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [02/27/2024 11:14:24.428] [test-akka.actor.default-dispatcher-15] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower560]: org.opendaylight.controller.cluster.common.actor.Monitor@370f1d8f 11:14:24,429 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower560 with journal batch size 1000 11:14:24,430 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: SnapshotOffer called. 11:14:24,430 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: Moving last applied index from -1 to -1 11:14:24,430 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: Set currentTerm=1, votedFor=null 11:14:24,430 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: Updated server config: isVoting: true, peers: [PeerInfo [id=leader559, address=akka://test/user/leader559, votingState=VOTING], PeerInfo [id=follower562, address=akka://test/user/follower562, votingState=NON_VOTING], PeerInfo [id=follower561, address=akka://test/user/follower561, votingState=VOTING]] 11:14:24,431 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower560 in 163.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [02/27/2024 11:14:24.431] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower561]: org.opendaylight.controller.cluster.common.actor.Monitor@11380a4e 11:14:24,432 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower561 with journal batch size 1000 11:14:24,434 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561: SnapshotOffer called. 11:14:24,434 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561: Moving last applied index from -1 to -1 11:14:24,434 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561: Set currentTerm=1, votedFor=null 11:14:24,434 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: Recovery completed in 3.328 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:24,434 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561: Updated server config: isVoting: true, peers: [PeerInfo [id=leader559, address=akka://test/user/leader559, votingState=VOTING], PeerInfo [id=follower560, address=akka://test/user/follower560, votingState=VOTING], PeerInfo [id=follower562, address=akka://test/user/follower562, votingState=NON_VOTING]] 11:14:24,434 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower561 in 107.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:24,435 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower561: Recovery completed in 1.415 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:24.535] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower562]: org.opendaylight.controller.cluster.common.actor.Monitor@73ef38a9 11:14:24,536 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower562 with journal batch size 1000 11:14:24,537 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: SnapshotOffer called. 11:14:24,537 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Moving last applied index from -1 to -1 11:14:24,537 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Set currentTerm=1, votedFor=null 11:14:24,537 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Updated server config: isVoting: false, peers: [PeerInfo [id=leader559, address=akka://test/user/leader559, votingState=VOTING], PeerInfo [id=follower560, address=akka://test/user/follower560, votingState=VOTING], PeerInfo [id=follower561, address=akka://test/user/follower561, votingState=VOTING]] 11:14:24,538 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower562 in 114.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:24,539 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Recovery completed in 1.760 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:24.539] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader559]: org.opendaylight.controller.cluster.common.actor.Monitor@3ce718be 11:14:24,540 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader559 with journal batch size 1000 11:14:24,540 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: SnapshotOffer called. 11:14:24,540 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Moving last applied index from -1 to -1 11:14:24,540 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Set currentTerm=1, votedFor=null 11:14:24,540 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Updated server config: isVoting: true, peers: [PeerInfo [id=follower560, address=akka://test/user/follower560, votingState=VOTING], PeerInfo [id=follower562, address=akka://test/user/follower562, votingState=NON_VOTING], PeerInfo [id=follower561, address=akka://test/user/follower561, votingState=VOTING]] 11:14:24,540 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader559 in 55.08 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:24,542 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Recovery completed in 1.381 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:24,875 AM [test-akka.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:14:24,875 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.238:25520, Removed) 11:14:24,875 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Follower): Received ElectionTimeout - switching to Candidate 11:14:24,875 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Candidate): Election: Candidate has following voting peers: [follower560, follower561] 11:14:24,875 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Set currentTerm=2, votedFor=leader559 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Candidate): Starting new election term 2 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Candidate): Sending RequestVote [term=2, candidateId=leader559, lastLogIndex=-1, lastLogTerm=-1] to peer follower560 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Candidate): Sending RequestVote [term=2, candidateId=leader559, lastLogIndex=-1, lastLogTerm=-1] to peer follower561 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:14:24,876 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:24,876 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): Cluster state: [] 11:14:24,876 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader559, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:24,876 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader559, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:24,876 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: Set currentTerm=2, votedFor=null 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561 (Follower): Cluster state: [] 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader559, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower561 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader559, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561: Set currentTerm=2, votedFor=null 11:14:24,876 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): In requestVote: RequestVote [term=2, candidateId=leader559, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:24,876 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: Set currentTerm=2, votedFor=leader559 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561 (Follower): In requestVote: RequestVote [term=2, candidateId=leader559, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:24,876 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:24,876 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561: Set currentTerm=2, votedFor=leader559 11:14:24,877 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:24,878 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:14:24,878 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:14:24,878 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Election: Leader has following peers: [follower560, follower562, follower561] 11:14:24,878 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower560: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:24,878 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower562: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:24,879 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower560: Last sync leader does not match current leader leader559, need to catch up to -1 11:14:24,879 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower561: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:24,879 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Term 2 in "AppendEntries [leaderId=leader559, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:14:24,879 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Set currentTerm=2, votedFor=null 11:14:24,879 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:14:24,879 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower562: Last sync leader does not match current leader leader559, need to catch up to -1 11:14:24,879 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower560 updated: matchIndex: -1, nextIndex: 0 [WARN] [02/27/2024 11:14:24.879] [test-akka.actor.default-dispatcher-7] [akka://test/user/follower560] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2e56e7b5 11:14:24,879 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower560: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:24,879 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower561: Last sync leader does not match current leader leader559, need to catch up to -1 11:14:24,879 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower562 updated: matchIndex: -1, nextIndex: 0 11:14:24,879 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:24,879 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower561 updated: matchIndex: -1, nextIndex: 0 11:14:24,879 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower561: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:24.879] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower561] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6c8465b8 [WARN] [02/27/2024 11:14:24.880] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower562] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@77b681fd 11:14:24,894 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:14:24,894 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:14:24,895 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:14:24,895 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 11:14:24,895 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:24,895 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:24,895 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, payload: class org.opendaylight.controller.cluster.raft.MockRaftActorContext$MockPayload, isSendImmediate: true 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower560: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): sendAppendEntries: 0 is present for follower follower560 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Sending AppendEntries to follower follower560: AppendEntries [leaderId=leader559, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower562: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): sendAppendEntries: 0 is present for follower follower562 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Sending AppendEntries to follower follower562: AppendEntries [leaderId=leader559, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:24,896 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): handleAppendEntries: AppendEntries [leaderId=leader559, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower561: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): sendAppendEntries: 0 is present for follower follower561 11:14:24,896 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): Number of entries to be appended = 1 11:14:24,896 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Sending AppendEntries to follower follower561: AppendEntries [leaderId=leader559, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:24,896 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): Append entry to log zero 11:14:24,896 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): handleAppendEntries: AppendEntries [leaderId=leader559, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:24,896 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): Log size is now 1 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Number of entries to be appended = 1 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:14:24,896 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower560, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Append entry to log zero 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:24,896 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower560: Lagging 0 entries behind leader leader559 and reached -1 (of expected -1) 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Log size is now 1 11:14:24,896 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower560 updated: matchIndex: 0, nextIndex: 1 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower562, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 11:14:24,896 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower562: Lagging 0 entries behind leader leader559 and reached -1 (of expected -1) 11:14:24,896 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower560: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:24,897 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower562 updated: matchIndex: 0, nextIndex: 1 11:14:24,897 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower562: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:24,897 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:24,897 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 11:14:24,897 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 11:14:24,897 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Setting last applied to 0 11:14:24,897 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Moving last applied index from -1 to 0 11:14:24,897 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Applying state for log index 0 data zero 11:14:24,897 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: applyState called: zero 11:14:24,898 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Persisting ApplyJournalEntries with index=0 [WARN] [02/27/2024 11:14:24.898] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower560] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7f26d6e1 11:14:24,898 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [02/27/2024 11:14:24.898] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower562] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4cf7bd94 11:14:24,995 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): sendAppendEntries: 0 is present for follower follower561 11:14:24,995 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Sending AppendEntries to follower follower561: AppendEntries [leaderId=leader559, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:25,116 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): sendAppendEntries: 0 is present for follower follower561 11:14:25,116 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Sending AppendEntries to follower follower561: AppendEntries [leaderId=leader559, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 11:14:25,116 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): Commit index set to 0 11:14:25,116 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower560: Lagging 0 entries behind leader leader559 and reached 0 (of expected -1) 11:14:25,116 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:14:25,116 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560 (Follower): Setting last applied to 0 11:14:25,116 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: Moving last applied index from -1 to 0 11:14:25,116 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: Applying state for log index 0 data zero 11:14:25,116 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Commit index set to 0 11:14:25,116 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: applyState called: zero 11:14:25,116 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower562: Lagging 0 entries behind leader leader559 and reached 0 (of expected -1) 11:14:25,116 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:14:25,116 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Setting last applied to 0 11:14:25,116 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Moving last applied index from -1 to 0 11:14:25,116 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Applying state for log index 0 data zero 11:14:25,116 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: Persisting ApplyJournalEntries with index=0 11:14:25,117 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: applyState called: zero 11:14:25,117 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Persisting ApplyJournalEntries with index=0 11:14:25,147 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 11:14:25,147 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:14:25,147 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: onRequestLeadership RequestLeadership [requestedFollowerId=follower561, replyTo=Actor[akka://test/user/$c#1358717783]] 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: Initiating leader transfer 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader559: pauseLeader successfully completed - doing transfer 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Attempting to transfer leadership 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower560: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower562: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Checking sendAppendEntries for follower follower561: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): sendAppendEntries: 0 is present for follower follower561 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower562: Lagging 0 entries behind leader leader559 and reached 0 (of expected -1) 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: onLeaderTransitioning: LeaderTransitioning [leaderId=leader559] 11:14:25,147 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower561: onLeaderTransitioning: LeaderTransitioning [leaderId=leader559] 11:14:25,148 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower560: Lagging 0 entries behind leader leader559 and reached 0 (of expected -1) 11:14:25,148 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower560: onLeaderTransitioning: LeaderTransitioning [leaderId=leader559] 11:14:25,356 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower562: Lagging 0 entries behind leader leader559 and reached 0 (of expected -1) 11:14:25,356 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower560: Lagging 0 entries behind leader leader559 and reached 0 (of expected -1) 11:14:25,475 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader559: leader transfer aborted 11:14:25,475 AM [test-akka.actor.default-dispatcher-5] [WARN] RaftActorLeadershipTransferCohort - leader559: Failed to transfer leadership in 328.0 ms 11:14:25,475 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559: LeadershipTransfer request from follower561 failed 11:14:25,475 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader559 (Leader): Leadership transfer expired 11:14:25,476 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower560: Lagging 0 entries behind leader leader559 and reached 0 (of expected -1) 11:14:25,476 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower562: Lagging 0 entries behind leader leader559 and reached 0 (of expected -1) 11:14:25,500 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 11:14:25,501 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower560-notifier563#1710487966] 11:14:25,512 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower560-collector564#-411299808] 11:14:25,512 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower560] 11:14:25,513 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower561-notifier565#-166016213] 11:14:25,513 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower561-collector566#705818842] 11:14:25,513 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower561] 11:14:25,513 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower562-notifier567#1375562580] 11:14:25,513 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower562-collector568#937350893] 11:14:25,513 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower562] 11:14:25,514 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader559-notifier569#-68823836] 11:14:25,514 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader559-collector570#975994998] 11:14:25,514 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader559] 11:14:25,514 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$c#1358717783] 11:14:25,515 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 11:14:25,515 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [02/27/2024 11:14:25.517] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower572]: org.opendaylight.controller.cluster.common.actor.Monitor@1ba51334 11:14:25,518 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower572 with journal batch size 1000 11:14:25,519 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572: SnapshotOffer called. 11:14:25,519 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572: Moving last applied index from -1 to -1 11:14:25,519 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572: Set currentTerm=1, votedFor=null 11:14:25,519 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572: Updated server config: isVoting: true, peers: [PeerInfo [id=follower574, address=akka://test/user/follower574, votingState=NON_VOTING], PeerInfo [id=leader571, address=akka://test/user/leader571, votingState=VOTING], PeerInfo [id=follower573, address=akka://test/user/follower573, votingState=VOTING]] 11:14:25,519 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower572 in 102.4 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [02/27/2024 11:14:25.520] [test-akka.actor.default-dispatcher-15] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower573]: org.opendaylight.controller.cluster.common.actor.Monitor@5c8ed478 11:14:25,520 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower573 with journal batch size 1000 11:14:25,521 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower572: Recovery completed in 1.513 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:25,522 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573: SnapshotOffer called. 11:14:25,522 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573: Moving last applied index from -1 to -1 11:14:25,522 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573: Set currentTerm=1, votedFor=null 11:14:25,522 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573: Updated server config: isVoting: true, peers: [PeerInfo [id=follower574, address=akka://test/user/follower574, votingState=NON_VOTING], PeerInfo [id=leader571, address=akka://test/user/leader571, votingState=VOTING], PeerInfo [id=follower572, address=akka://test/user/follower572, votingState=VOTING]] 11:14:25,522 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower573 in 74.44 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:25,523 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower573: Recovery completed in 811.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [02/27/2024 11:14:25.622] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower574]: org.opendaylight.controller.cluster.common.actor.Monitor@2c5d7fb6 11:14:25,622 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower574 with journal batch size 1000 11:14:25,624 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: SnapshotOffer called. 11:14:25,624 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Moving last applied index from -1 to -1 11:14:25,624 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Set currentTerm=1, votedFor=null 11:14:25,624 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Updated server config: isVoting: false, peers: [PeerInfo [id=leader571, address=akka://test/user/leader571, votingState=VOTING], PeerInfo [id=follower573, address=akka://test/user/follower573, votingState=VOTING], PeerInfo [id=follower572, address=akka://test/user/follower572, votingState=VOTING]] 11:14:25,624 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower574 in 82.30 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [02/27/2024 11:14:25.625] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader571]: org.opendaylight.controller.cluster.common.actor.Monitor@5700ed72 11:14:25,625 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader571 with journal batch size 1000 11:14:25,625 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Recovery completed in 1.176 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:25,626 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: SnapshotOffer called. 11:14:25,626 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: Moving last applied index from -1 to -1 11:14:25,626 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: Set currentTerm=1, votedFor=null 11:14:25,626 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: Updated server config: isVoting: true, peers: [PeerInfo [id=follower574, address=akka://test/user/follower574, votingState=NON_VOTING], PeerInfo [id=follower573, address=akka://test/user/follower573, votingState=VOTING], PeerInfo [id=follower572, address=akka://test/user/follower572, votingState=VOTING]] 11:14:25,626 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader571 in 64.15 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:14:25,634 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: Recovery completed in 7.610 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:14:25,996 AM [test-akka.actor.default-dispatcher-7] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:14:25,996 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.238:25520, Removed) 11:14:25,996 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Follower): Received ElectionTimeout - switching to Candidate 11:14:25,996 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Candidate): Election: Candidate has following voting peers: [follower573, follower572] 11:14:25,996 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: Set currentTerm=2, votedFor=leader571 11:14:25,996 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Candidate): Starting new election term 2 11:14:25,996 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Candidate): Sending RequestVote [term=2, candidateId=leader571, lastLogIndex=-1, lastLogTerm=-1] to peer follower573 11:14:25,997 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Candidate): Sending RequestVote [term=2, candidateId=leader571, lastLogIndex=-1, lastLogTerm=-1] to peer follower572 11:14:25,997 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:14:25,997 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:25,997 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Follower): Cluster state: [] 11:14:25,997 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:25,997 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader571, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:25,997 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572 (Follower): Cluster state: [] 11:14:25,997 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader571, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:25,997 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader571, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:25,997 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower572 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader571, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 11:14:25,997 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573: Set currentTerm=2, votedFor=null 11:14:25,997 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572: Set currentTerm=2, votedFor=null 11:14:25,997 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572 (Follower): In requestVote: RequestVote [term=2, candidateId=leader571, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:25,997 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572: Set currentTerm=2, votedFor=leader571 11:14:25,997 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Follower): In requestVote: RequestVote [term=2, candidateId=leader571, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:25,997 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573: Set currentTerm=2, votedFor=leader571 11:14:25,997 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower572 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:25,997 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 11:14:25,999 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 11:14:25,999 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 11:14:25,999 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Election: Leader has following peers: [follower574, follower573, follower572] 11:14:25,999 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower574: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:25,999 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower573: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:25,999 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower572: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:25,999 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:14:25,999 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower574 (Follower): Term 2 in "AppendEntries [leaderId=leader571, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 11:14:25,999 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Set currentTerm=2, votedFor=null 11:14:26,0 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower574: Last sync leader does not match current leader leader571, need to catch up to -1 11:14:26,0 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower574 updated: matchIndex: -1, nextIndex: 0 11:14:26,0 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower574: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:26,0 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower573: Last sync leader does not match current leader leader571, need to catch up to -1 11:14:26,0 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower572: Last sync leader does not match current leader leader571, need to catch up to -1 11:14:26,0 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower573 updated: matchIndex: -1, nextIndex: 0 11:14:26,0 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower573: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:26,0 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower572 updated: matchIndex: -1, nextIndex: 0 11:14:26,0 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower572: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:26.000] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower573] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@14ac2a4d [WARN] [02/27/2024 11:14:26.000] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower572] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7b35b09c [WARN] [02/27/2024 11:14:26.000] [test-akka.actor.default-dispatcher-7] [akka://test/user/follower574] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@167c7508 11:14:26,37 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:14:26,37 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:14:26,37 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:14:26,38 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower572] 11:14:26,49 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower574] 11:14:26,59 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:14:26,59 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:14:26,59 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: onRequestLeadership RequestLeadership [requestedFollowerId=follower573, replyTo=Actor[akka://test/user/$d#-133453587]] 11:14:26,59 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: Initiating leader transfer 11:14:26,60 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573: onLeaderTransitioning: LeaderTransitioning [leaderId=leader571] 11:14:26,60 AM [test-akka.actor.default-dispatcher-7] [DEBUG] RaftActorLeadershipTransferCohort - leader571: pauseLeader successfully completed - doing transfer 11:14:26,60 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Attempting to transfer leadership [WARN] [02/27/2024 11:14:26.060] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower574] received dead letter from TestActor[akka://test/user/leader571]: LeaderTransitioning [leaderId=leader571] [WARN] [02/27/2024 11:14:26.060] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower572] received dead letter from TestActor[akka://test/user/leader571]: LeaderTransitioning [leaderId=leader571] 11:14:26,60 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower574: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:26,60 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower573: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:26,60 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower572: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:26.060] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower574] received dead letter from TestActor[akka://test/user/leader571]: AppendEntries [leaderId=leader571, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:14:26,60 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower573: Lagging 0 entries behind leader leader571 and reached -1 (of expected -1) [WARN] [02/27/2024 11:14:26.060] [test-akka.actor.default-dispatcher-7] [akka://test/user/follower572] received dead letter from TestActor[akka://test/user/leader571]: AppendEntries [leaderId=leader571, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:14:26,60 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): tryToCompleteLeadershipTransfer: followerId: follower573, matchIndex: -1, lastIndex: -1, isVoting: true 11:14:26,60 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Follower's log matches - sending ElectionTimeout 11:14:26,60 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower574: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:26.060] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower573] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7031c5ca 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower573: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Checking sendAppendEntries for follower follower572: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:26.061] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower574] received dead letter from TestActor[akka://test/user/leader571]: AppendEntries [leaderId=leader571, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Leader transfer complete 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader571: leader transfer complete - waiting for new leader [WARN] [02/27/2024 11:14:26.061] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower572] received dead letter from TestActor[akka://test/user/leader571]: AppendEntries [leaderId=leader571, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower573: Lagging 0 entries behind leader leader571 and reached -1 (of expected -1) 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Follower): Received TimeoutNow - switching to Candidate 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Candidate): Election: Candidate has following voting peers: [leader571, follower572] 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573: Set currentTerm=3, votedFor=follower573 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Candidate): Starting new election term 3 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Candidate): Sending RequestVote [term=3, candidateId=follower573, lastLogIndex=-1, lastLogTerm=-1] to peer leader571 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Candidate): Sending RequestVote [term=3, candidateId=follower573, lastLogIndex=-1, lastLogTerm=-1] to peer follower572 11:14:26,61 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:14:26,61 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Cluster state: [] 11:14:26,61 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 11:14:26,61 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower573, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 11:14:26,61 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower573, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 11:14:26,61 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: Set currentTerm=3, votedFor=null 11:14:26,61 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): Leadership transfer in progress - processing RequestVote 11:14:26,62 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): In requestVote: RequestVote [term=3, candidateId=follower573, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:14:26,62 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: Set currentTerm=3, votedFor=follower573 11:14:26,62 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] [WARN] [02/27/2024 11:14:26.062] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower572] received dead letter from TestActor[akka://test/user/follower573]: RequestVote [term=3, candidateId=follower573, lastLogIndex=-1, lastLogTerm=-1] 11:14:26,62 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader571 (Leader) :- Switching from behavior Leader to Follower, election term: 3 11:14:26,65 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 11:14:26,65 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 11:14:26,65 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Leader): Election: Leader has following peers: [follower574, leader571, follower572] 11:14:26,65 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Leader): Checking sendAppendEntries for follower follower574: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:26,65 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Leader): Checking sendAppendEntries for follower leader571: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:26,65 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Leader): Checking sendAppendEntries for follower follower572: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:14:26,65 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 11:14:26,65 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - leader571: Last sync leader does not match current leader follower573, need to catch up to -1 11:14:26,65 AM [test-akka.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader571: leader changed to follower573 11:14:26,66 AM [test-akka.actor.default-dispatcher-18] [INFO] RaftActorLeadershipTransferCohort - leader571: Successfully transferred leadership to follower573 in 5.982 ms 11:14:26,66 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader571: Leadership transferred successfully to follower573 [WARN] [02/27/2024 11:14:26.066] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower574] received dead letter from TestActor[akka://test/user/follower573]: AppendEntries [leaderId=follower573, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [02/27/2024 11:14:26.066] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower572] received dead letter from TestActor[akka://test/user/follower573]: AppendEntries [leaderId=follower573, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 11:14:26,66 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader571 updated: matchIndex: -1, nextIndex: 0 11:14:26,66 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower573 (Leader): Checking sendAppendEntries for follower leader571: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [02/27/2024 11:14:26.066] [test-akka.actor.default-dispatcher-18] [akka://test/user/leader571] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3446466 11:14:26,111 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 11:14:26,111 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower572-notifier575#1692341112] 11:14:26,122 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower572-collector576#1216460359] 11:14:26,122 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower573-notifier577#-1852076061] 11:14:26,122 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower573-collector578#-1107608967] 11:14:26,123 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower573] 11:14:26,123 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower574-notifier579#1304420074] 11:14:26,123 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower574-collector580#-981648977] 11:14:26,123 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader571-notifier581#-763393902] 11:14:26,123 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader571-collector582#811727835] 11:14:26,123 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader571] 11:14:26,123 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$d#-133453587] [INFO] [02/27/2024 11:14:26.125] [main] [CoordinatedShutdown(akka://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 11:14:26,125 AM [test-akka.actor.default-dispatcher-16] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [02/27/2024 11:14:26.126] [test-akka.actor.default-dispatcher-18] [akka://test/system/cluster/core/daemon] unhandled message from Actor[akka://test/deadLetters]: Leave(akka://test@10.30.171.238:25520) [INFO] [02/27/2024 11:14:26.127] [test-akka.actor.internal-dispatcher-9] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.238:25520] - Exiting completed 11:14:26,127 AM [test-akka.actor.default-dispatcher-16] [INFO] Cluster - Cluster Node [akka://test@10.30.171.238:25520] - Exiting completed 11:14:26,127 AM [test-akka.actor.default-dispatcher-16] [INFO] Cluster - Cluster Node [akka://test@10.30.171.238:25520] - Shutting down... [INFO] [02/27/2024 11:14:26.127] [test-akka.actor.internal-dispatcher-9] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.238:25520] - Shutting down... [INFO] [02/27/2024 11:14:26.127] [test-akka.actor.internal-dispatcher-9] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.238:25520] - Successfully shut down 11:14:26,127 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@10.30.171.238:25520] - Successfully shut down [INFO] [02/27/2024 11:14:26.130] [test-akka.remote.default-remote-dispatcher-8] [akka://test@10.30.171.238:25520/system/remoting-terminator] Shutting down remote daemon. 11:14:26,130 AM [test-akka.actor.default-dispatcher-18] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. [INFO] [02/27/2024 11:14:26.130] [test-akka.remote.default-remote-dispatcher-8] [akka://test@10.30.171.238:25520/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 11:14:26,130 AM [test-akka.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports.