[DEBUG] [10/18/2024 01:12:38.106] [main] [EventStream(akka://test)] logger log1-TestEventListener started 01:12:38,107 AM [test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [10/18/2024 01:12:38.107] [main] [EventStream(akka://test)] logger log2-Slf4jLogger started [DEBUG] [10/18/2024 01:12:38.107] [main] [EventStream(akka://test)] Default Loggers started [INFO] [10/18/2024 01:12:38.118] [main] [ArteryTransport(akka://test)] Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.171.33:25520] with UID [6746490320898564491] 01:12:38,118 AM [test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.171.33:25520] with UID [6746490320898564491] 01:12:38,119 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.33:25520] - Starting up, Akka version [2.6.21] ... [INFO] [10/18/2024 01:12:38.119] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.33:25520] - Starting up, Akka version [2.6.21] ... [INFO] [10/18/2024 01:12:38.121] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.33:25520] - Registered cluster JMX MBean [akka:type=Cluster] [INFO] [10/18/2024 01:12:38.121] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.33:25520] - Started up successfully 01:12:38,121 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.33:25520] - Registered cluster JMX MBean [akka:type=Cluster] 01:12:38,121 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.33:25520] - Started up successfully 01:12:38,125 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.33:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 01:12:38,125 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.33:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining [INFO] [10/18/2024 01:12:38.122] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.33:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing [INFO] [10/18/2024 01:12:38.122] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.33:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 01:12:38,131 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 01:12:38,131 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/18/2024 01:12:38.133] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower551]: org.opendaylight.controller.cluster.common.actor.Monitor@3ec7831f [INFO] [akkaDeadLetter][10/18/2024 01:12:38.134] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower551] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,134 AM [test-akka.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower551] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,135 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower551 with journal batch size 1000 01:12:38,136 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: SnapshotOffer called. 01:12:38,136 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Moving last applied index from -1 to -1 01:12:38,136 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=1, votedFor=null 01:12:38,136 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=akka://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=akka://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=akka://test/user/leader550, votingState=VOTING]] 01:12:38,136 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower551 in 122.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:38,137 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower552 with journal batch size 1000 [WARN] [10/18/2024 01:12:38.137] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower552]: org.opendaylight.controller.cluster.common.actor.Monitor@46dc1729 [INFO] [akkaDeadLetter][10/18/2024 01:12:38.137] [test-akka.actor.default-dispatcher-13] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower552] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,137 AM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower552] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,138 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Recovery completed in 2.434 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:38,140 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: SnapshotOffer called. 01:12:38,140 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Moving last applied index from -1 to -1 01:12:38,140 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=1, votedFor=null 01:12:38,140 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=akka://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=leader550, address=akka://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=akka://test/user/follower551, votingState=VOTING]] 01:12:38,140 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower552 in 107.0 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:38,141 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Recovery completed in 725.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:38.239] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower553]: org.opendaylight.controller.cluster.common.actor.Monitor@56f013ab [INFO] [akkaDeadLetter][10/18/2024 01:12:38.240] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower553] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,240 AM [test-akka.actor.default-dispatcher-16] [INFO] EmptyLocalActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower553] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,240 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower553 with journal batch size 1000 01:12:38,241 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: SnapshotOffer called. 01:12:38,241 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Moving last applied index from -1 to -1 01:12:38,241 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Set currentTerm=1, votedFor=null 01:12:38,241 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Updated server config: isVoting: false, peers: [PeerInfo [id=follower552, address=akka://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=akka://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=akka://test/user/follower551, votingState=VOTING]] 01:12:38,242 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower553 in 143.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:38,243 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Recovery completed in 1.329 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:38.341] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader550]: org.opendaylight.controller.cluster.common.actor.Monitor@1fa49ab7 [INFO] [akkaDeadLetter][10/18/2024 01:12:38.341] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader550] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,341 AM [test-akka.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader550] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,342 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader550 with journal batch size 1000 01:12:38,343 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: SnapshotOffer called. 01:12:38,343 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Moving last applied index from -1 to -1 01:12:38,343 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Set currentTerm=1, votedFor=null 01:12:38,343 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=akka://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=akka://test/user/follower552, votingState=VOTING], PeerInfo [id=follower551, address=akka://test/user/follower551, votingState=VOTING]] 01:12:38,344 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader550 in 151.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:38,345 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Recovery completed in 1.235 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:38,703 AM [test-akka.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:12:38,703 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.33:25520, Removed) 01:12:38,704 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Received ElectionTimeout - switching to Candidate 01:12:38,704 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Election: Candidate has following voting peers: [follower552, follower551] 01:12:38,704 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Set currentTerm=2, votedFor=leader550 01:12:38,704 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Starting new election term 2 01:12:38,704 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] to peer follower552 01:12:38,704 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] to peer follower551 01:12:38,704 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:38,704 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:12:38,705 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Cluster state: [] 01:12:38,705 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:38,705 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:38,705 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:38,705 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Cluster state: [] 01:12:38,705 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=2, votedFor=null 01:12:38,705 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:38,705 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:38,705 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=2, votedFor=null 01:12:38,705 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): In requestVote: RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:38,705 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=2, votedFor=leader550 01:12:38,705 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:38,705 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): In requestVote: RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:38,705 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=2, votedFor=leader550 01:12:38,705 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:38,707 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:12:38,707 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:12:38,707 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Election: Leader has following peers: [follower553, follower552, follower551] 01:12:38,707 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:38,707 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:38,707 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:38,707 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:12:38,707 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553 (Follower): Term 2 in "AppendEntries [leaderId=leader550, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:12:38,707 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Set currentTerm=2, votedFor=null 01:12:38,707 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower552: Last sync leader does not match current leader leader550, need to catch up to -1 01:12:38,707 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower553: Last sync leader does not match current leader leader550, need to catch up to -1 01:12:38,708 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower551: Last sync leader does not match current leader leader550, need to catch up to -1 01:12:38,708 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower552 updated: matchIndex: -1, nextIndex: 0 01:12:38,708 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:38.708] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower552] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@37993e58 01:12:38,708 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower553 updated: matchIndex: -1, nextIndex: 0 [WARN] [10/18/2024 01:12:38.708] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower551] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@608ef733 01:12:38,708 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:38,708 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower551 updated: matchIndex: -1, nextIndex: 0 01:12:38,708 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [INFO] [akkaDeadLetter][10/18/2024 01:12:38.708] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower552] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,708 AM [test-akka.actor.default-dispatcher-13] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower552] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][10/18/2024 01:12:38.708] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower551] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,708 AM [test-akka.actor.default-dispatcher-13] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower551] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,709 AM [test-akka.actor.default-dispatcher-18] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower553] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [10/18/2024 01:12:38.709] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower553] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4a38be4a [INFO] [akkaDeadLetter][10/18/2024 01:12:38.709] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower553] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,748 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:12:38,748 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader550 starting 01:12:38,750 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: onShutDown 01:12:38,750 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Initiating leader transfer 01:12:38,751 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 01:12:38,751 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader550: pauseLeader successfully completed - doing transfer 01:12:38,751 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 01:12:38,751 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Attempting to transfer leadership 01:12:38,751 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:38,751 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:38,751 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:38,751 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:12:38,751 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [10/18/2024 01:12:38.751] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower553] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3b518f66 01:12:38,751 AM [test-akka.actor.default-dispatcher-11] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower553] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [10/18/2024 01:12:38.751] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower551] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3d297bc9 01:12:38,751 AM [test-akka.actor.default-dispatcher-11] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower551] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][10/18/2024 01:12:38.751] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower553] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][10/18/2024 01:12:38.751] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower551] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,751 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 01:12:38,752 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [10/18/2024 01:12:38.752] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower552] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@22ba2231 01:12:38,752 AM [test-akka.actor.default-dispatcher-12] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower552] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][10/18/2024 01:12:38.752] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower552] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:12:38,823 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:12:38,824 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:12:38,824 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:12:38,943 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:12:38,943 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:12:38,943 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:12:39,64 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader550: leader transfer aborted 01:12:39,65 AM [test-akka.actor.default-dispatcher-5] [WARN] RaftActorLeadershipTransferCohort - leader550: Failed to transfer leadership in 314.2 ms 01:12:39,65 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: leader transfer failed - sending PoisonPill 01:12:39,65 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Leadership transfer expired 01:12:39,65 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:12:39,65 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:12:39,65 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [10/18/2024 01:12:39.067] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader550] received dead letter from TestActor[akka://test/user/follower552]: AppendEntriesReply [term=2, success=true, followerId=follower552, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [10/18/2024 01:12:39.067] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader550] received dead letter from TestActor[akka://test/user/follower553]: AppendEntriesReply [term=2, success=true, followerId=follower553, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [10/18/2024 01:12:39.067] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader550] received dead letter from TestActor[akka://test/user/follower551]: AppendEntriesReply [term=2, success=true, followerId=follower551, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:12:39,68 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader550 ending 01:12:39,69 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 01:12:39,69 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower551-notifier554#1459974820] 01:12:39,81 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower551-collector555#-1065996615] 01:12:39,81 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower551] 01:12:39,82 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower552-notifier556#-1805928537] 01:12:39,82 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower552-collector557#-1961536063] 01:12:39,82 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower552] 01:12:39,83 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower553-notifier558#-1893676188] 01:12:39,83 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower553-collector559#-1447763869] 01:12:39,83 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower553] 01:12:39,83 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader550-notifier560#558252064] 01:12:39,84 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader550-collector561#1359025087] 01:12:39,84 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader550] 01:12:39,85 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 01:12:39,85 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/18/2024 01:12:39.088] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower563]: org.opendaylight.controller.cluster.common.actor.Monitor@64d6703d 01:12:39,88 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower563 with journal batch size 1000 01:12:39,89 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: SnapshotOffer called. 01:12:39,89 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Moving last applied index from -1 to -1 01:12:39,89 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=1, votedFor=null 01:12:39,89 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=akka://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=akka://test/user/follower564, votingState=VOTING], PeerInfo [id=follower565, address=akka://test/user/follower565, votingState=NON_VOTING]] 01:12:39,89 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower563 in 142.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:39,90 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Recovery completed in 1.253 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:39.091] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower564]: org.opendaylight.controller.cluster.common.actor.Monitor@433cf9ca 01:12:39,91 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower564 with journal batch size 1000 01:12:39,92 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: SnapshotOffer called. 01:12:39,92 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Moving last applied index from -1 to -1 01:12:39,92 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=1, votedFor=null [WARN] [10/18/2024 01:12:39.093] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower565]: org.opendaylight.controller.cluster.common.actor.Monitor@2685b2be 01:12:39,93 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower565 with journal batch size 1000 01:12:39,94 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: SnapshotOffer called. 01:12:39,94 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Moving last applied index from -1 to -1 01:12:39,94 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=1, votedFor=null 01:12:39,94 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Updated server config: isVoting: false, peers: [PeerInfo [id=leader562, address=akka://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=akka://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=akka://test/user/follower563, votingState=VOTING]] 01:12:39,94 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower565 in 96.18 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:39,95 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Recovery completed in 824.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:39.095] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader562]: org.opendaylight.controller.cluster.common.actor.Monitor@2086d38d 01:12:39,95 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader562 with journal batch size 1000 01:12:39,96 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: SnapshotOffer called. 01:12:39,96 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Moving last applied index from -1 to -1 01:12:39,96 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=1, votedFor=null 01:12:39,96 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Updated server config: isVoting: true, peers: [PeerInfo [id=follower564, address=akka://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=akka://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=akka://test/user/follower565, votingState=NON_VOTING]] 01:12:39,96 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader562 in 95.82 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:39,96 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Recovery completed in 691.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:39,98 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=akka://test/user/leader562, votingState=VOTING], PeerInfo [id=follower563, address=akka://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=akka://test/user/follower565, votingState=NON_VOTING]] 01:12:39,98 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower564 in 6.050 ms: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:39,99 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Recovery completed in 6.983 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:39,483 AM [test-akka.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:12:39,483 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.33:25520, Removed) 01:12:39,483 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Received ElectionTimeout - switching to Candidate 01:12:39,483 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Election: Candidate has following voting peers: [follower564, follower563] 01:12:39,483 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=2, votedFor=leader562 01:12:39,483 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Starting new election term 2 01:12:39,483 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] to peer follower564 01:12:39,483 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] to peer follower563 01:12:39,483 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:39,483 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:12:39,483 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Cluster state: [] 01:12:39,483 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:39,483 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:39,483 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=2, votedFor=null 01:12:39,484 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): In requestVote: RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:39,484 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=2, votedFor=leader562 01:12:39,484 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:39,484 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:39,484 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 01:12:39,484 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:39,484 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:39,484 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=2, votedFor=null 01:12:39,484 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:39,484 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=2, votedFor=leader562 01:12:39,484 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:39,485 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:12:39,485 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:12:39,485 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Election: Leader has following peers: [follower564, follower563, follower565] 01:12:39,485 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,485 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,486 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader leader562, need to catch up to -1 01:12:39,486 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower564: Last sync leader does not match current leader leader562, need to catch up to -1 01:12:39,486 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,486 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 [WARN] [10/18/2024 01:12:39.486] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower563] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1baa51e0 01:12:39,486 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 2 in "AppendEntries [leaderId=leader562, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:12:39,486 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=2, votedFor=null [WARN] [10/18/2024 01:12:39.486] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower564] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@46824c4e 01:12:39,486 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower564 updated: matchIndex: -1, nextIndex: 0 01:12:39,486 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,486 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader leader562, need to catch up to -1 01:12:39,486 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 01:12:39,486 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,486 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 01:12:39,486 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:39.487] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower565] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@68544a9b 01:12:39,499 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:12:39,499 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:12:39,500 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:12:39,500 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:12:39,501 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:12:39,501 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: onRequestLeadership RequestLeadership [requestedFollowerId=follower564, replyTo=Actor[akka://test/user/$a#-1483058178]] 01:12:39,502 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Initiating leader transfer 01:12:39,502 AM [test-akka.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader562: pauseLeader successfully completed - doing transfer 01:12:39,502 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Attempting to transfer leadership 01:12:39,502 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,503 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,503 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,503 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:12:39,503 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:12:39,503 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 01:12:39,503 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] [WARN] [10/18/2024 01:12:39.503] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower563] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1e04e471 [WARN] [10/18/2024 01:12:39.503] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower565] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@448c62ce 01:12:39,503 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:12:39,503 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 01:12:39,504 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): tryToCompleteLeadershipTransfer: followerId: follower564, matchIndex: -1, lastIndex: -1, isVoting: true 01:12:39,504 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Follower's log matches - sending ElectionTimeout 01:12:39,504 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,504 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,504 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:39.504] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower564] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@70bbf9e5 01:12:39,504 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leader transfer complete 01:12:39,504 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader transfer complete - waiting for new leader 01:12:39,504 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:12:39,504 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:12:39,504 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:12:39,504 AM [test-akka.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:12:39,504 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Received TimeoutNow - switching to Candidate 01:12:39,504 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Election: Candidate has following voting peers: [leader562, follower563] 01:12:39,504 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=3, votedFor=follower564 01:12:39,504 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Starting new election term 3 01:12:39,504 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] to peer leader562 01:12:39,505 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] to peer follower563 01:12:39,505 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:12:39,505 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:39,505 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 01:12:39,505 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:39,505 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 2 - updating term 01:12:39,505 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=3, votedFor=null 01:12:39,505 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:39,506 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:39,506 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=3, votedFor=follower564 01:12:39,506 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Cluster state: [] 01:12:39,506 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:12:39,506 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:39,506 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 01:12:39,506 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=3, votedFor=null 01:12:39,506 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leadership transfer in progress - processing RequestVote 01:12:39,506 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): In requestVote: RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:39,506 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=3, votedFor=follower564 01:12:39,506 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 01:12:39,506 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 01:12:39,506 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:12:39,506 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Election: Leader has following peers: [leader562, follower563, follower565] 01:12:39,506 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader) :- Switching from behavior Leader to Follower, election term: 3 01:12:39,506 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,507 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,507 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,507 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 01:12:39,507 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 3 in "AppendEntries [leaderId=follower564, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 01:12:39,507 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=3, votedFor=null 01:12:39,507 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader follower564, need to catch up to -1 01:12:39,507 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader follower564, need to catch up to -1 01:12:39,507 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - leader562: Last sync leader does not match current leader follower564, need to catch up to -1 01:12:39,507 AM [test-akka.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader changed to follower564 [WARN] [10/18/2024 01:12:39.507] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower563] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@58775071 01:12:39,507 AM [test-akka.actor.default-dispatcher-11] [INFO] RaftActorLeadershipTransferCohort - leader562: Successfully transferred leadership to follower564 in 5.130 ms 01:12:39,508 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Leadership transferred successfully to follower564 01:12:39,508 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 [WARN] [10/18/2024 01:12:39.508] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower565] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5e94a2a 01:12:39,508 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,508 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 01:12:39,508 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:39,508 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader562 updated: matchIndex: -1, nextIndex: 0 01:12:39,508 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:39.508] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader562] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3b33c6d0 01:12:39,552 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 01:12:39,552 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower563-notifier566#635613548] 01:12:39,563 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower563-collector567#197671901] 01:12:39,563 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower563] 01:12:39,564 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower564-notifier568#-1796161914] 01:12:39,564 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower564-collector569#-69268796] 01:12:39,564 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower564] 01:12:39,564 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower565-notifier570#1955590297] 01:12:39,564 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower565-collector571#291930175] 01:12:39,565 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower565] 01:12:39,565 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader562-notifier572#-1524525027] 01:12:39,565 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader562-collector573#1315224878] 01:12:39,565 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader562] 01:12:39,565 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$a#-1483058178] 01:12:39,566 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 01:12:39,566 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/18/2024 01:12:39.569] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower575]: org.opendaylight.controller.cluster.common.actor.Monitor@1a97c3ba 01:12:39,570 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower575 with journal batch size 1000 01:12:39,572 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: SnapshotOffer called. 01:12:39,572 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Moving last applied index from -1 to -1 01:12:39,572 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=1, votedFor=null 01:12:39,572 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=akka://test/user/leader574, votingState=VOTING], PeerInfo [id=follower577, address=akka://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=akka://test/user/follower576, votingState=VOTING]] 01:12:39,572 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower575 in 172.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:39,573 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Recovery completed in 1.637 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:39.671] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower576]: org.opendaylight.controller.cluster.common.actor.Monitor@8a7cf92 01:12:39,672 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower576 with journal batch size 1000 01:12:39,672 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: SnapshotOffer called. 01:12:39,672 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Moving last applied index from -1 to -1 01:12:39,672 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=1, votedFor=null 01:12:39,672 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=akka://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=akka://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=akka://test/user/follower577, votingState=NON_VOTING]] 01:12:39,672 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower576 in 129.0 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:39,674 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Recovery completed in 1.280 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:39.773] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower577]: org.opendaylight.controller.cluster.common.actor.Monitor@7fcc95d7 01:12:39,773 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower577 with journal batch size 1000 01:12:39,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: SnapshotOffer called. 01:12:39,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Moving last applied index from -1 to -1 01:12:39,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Set currentTerm=1, votedFor=null 01:12:39,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Updated server config: isVoting: false, peers: [PeerInfo [id=leader574, address=akka://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=akka://test/user/follower575, votingState=VOTING], PeerInfo [id=follower576, address=akka://test/user/follower576, votingState=VOTING]] 01:12:39,774 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower577 in 65.57 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:39,775 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Recovery completed in 1.071 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:39.776] [test-akka.actor.default-dispatcher-13] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader574]: org.opendaylight.controller.cluster.common.actor.Monitor@38b70148 01:12:39,776 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader574 with journal batch size 1000 01:12:39,777 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: SnapshotOffer called. 01:12:39,777 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Moving last applied index from -1 to -1 01:12:39,777 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Set currentTerm=1, votedFor=null 01:12:39,777 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Updated server config: isVoting: true, peers: [PeerInfo [id=follower575, address=akka://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=akka://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=akka://test/user/follower576, votingState=VOTING]] 01:12:39,777 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader574 in 63.41 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:39,778 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Recovery completed in 769.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:40,193 AM [test-akka.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:12:40,193 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.33:25520, Removed) 01:12:40,193 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Received ElectionTimeout - switching to Candidate 01:12:40,193 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Election: Candidate has following voting peers: [follower575, follower576] 01:12:40,193 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Set currentTerm=2, votedFor=leader574 01:12:40,193 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Starting new election term 2 01:12:40,193 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] to peer follower575 01:12:40,194 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] to peer follower576 01:12:40,194 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:40,194 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:40,194 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Cluster state: [] 01:12:40,194 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:40,194 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Cluster state: [] 01:12:40,194 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:40,194 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=2, votedFor=null 01:12:40,194 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:40,194 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:40,194 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=2, votedFor=null 01:12:40,194 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): In requestVote: RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:40,194 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): In requestVote: RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:40,194 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=2, votedFor=leader574 01:12:40,194 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=2, votedFor=leader574 01:12:40,194 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:40,194 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:40,194 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:12:40,197 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:12:40,197 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:12:40,197 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Election: Leader has following peers: [follower575, follower577, follower576] 01:12:40,197 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:40,198 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:40,198 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:40,198 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:12:40,198 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577 (Follower): Term 2 in "AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:12:40,198 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Set currentTerm=2, votedFor=null 01:12:40,198 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower576: Last sync leader does not match current leader leader574, need to catch up to -1 01:12:40,198 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower577: Last sync leader does not match current leader leader574, need to catch up to -1 01:12:40,198 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower575: Last sync leader does not match current leader leader574, need to catch up to -1 01:12:40,198 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower576 updated: matchIndex: -1, nextIndex: 0 01:12:40,198 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:40,198 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower575 updated: matchIndex: -1, nextIndex: 0 01:12:40,198 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:40.198] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower576] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@36eb1907 01:12:40,198 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower577 updated: matchIndex: -1, nextIndex: 0 01:12:40,198 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:40.199] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower575] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2f26036c [WARN] [10/18/2024 01:12:40.199] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower577] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7da3bc 01:12:40,231 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:12:40,231 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:12:40,232 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:12:40,232 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower576 starting 01:12:40,232 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: onShutDown 01:12:40,233 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower576 ending 01:12:40,233 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:12:40,233 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:12:40,233 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: onRequestLeadership RequestLeadership [requestedFollowerId=follower576, replyTo=Actor[akka://test/user/$b#-625265608]] 01:12:40,233 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Initiating leader transfer 01:12:40,233 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] 01:12:40,233 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] 01:12:40,233 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader574: pauseLeader successfully completed - doing transfer 01:12:40,233 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Attempting to transfer leadership [WARN] [10/18/2024 01:12:40.233] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower576] received dead letter from TestActor[akka://test/user/leader574]: LeaderTransitioning [leaderId=leader574] 01:12:40,233 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:40,233 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:40,233 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:40,233 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [10/18/2024 01:12:40.233] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower576] received dead letter from TestActor[akka://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [10/18/2024 01:12:40.234] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower575] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@78a496aa 01:12:40,234 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [10/18/2024 01:12:40.234] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower577] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@c891b87 [WARN] [10/18/2024 01:12:40.313] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower576] received dead letter from TestActor[akka://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:12:40,434 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 01:12:40,434 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [10/18/2024 01:12:40.434] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower576] received dead letter from TestActor[akka://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:12:40,553 AM [test-akka.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader574: leader transfer aborted 01:12:40,553 AM [test-akka.actor.default-dispatcher-12] [WARN] RaftActorLeadershipTransferCohort - leader574: Failed to transfer leadership in 320.0 ms 01:12:40,553 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: LeadershipTransfer request from follower576 failed 01:12:40,554 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Leadership transfer expired 01:12:40,555 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 01:12:40,555 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [10/18/2024 01:12:40.555] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower576] received dead letter from TestActor[akka://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:12:40,586 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 01:12:40,586 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower575-notifier578#1949802228] 01:12:40,597 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower575-collector579#2074680150] 01:12:40,598 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower575] 01:12:40,598 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower576-notifier580#1914227650] 01:12:40,598 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower576-collector581#904879269] 01:12:40,598 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower576] 01:12:40,598 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower577-notifier582#-1246147312] 01:12:40,598 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower577-collector583#-1548884367] 01:12:40,599 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower577] 01:12:40,599 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader574-notifier584#506514615] 01:12:40,599 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader574-collector585#701448972] 01:12:40,599 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader574] 01:12:40,599 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$b#-625265608] 01:12:40,600 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 01:12:40,600 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/18/2024 01:12:40.602] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower587]: org.opendaylight.controller.cluster.common.actor.Monitor@5430cf6d 01:12:40,603 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower587 with journal batch size 1000 01:12:40,604 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: SnapshotOffer called. 01:12:40,604 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to -1 01:12:40,604 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=1, votedFor=null 01:12:40,604 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=akka://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=akka://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=akka://test/user/follower588, votingState=VOTING]] 01:12:40,604 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower587 in 109.0 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [10/18/2024 01:12:40.605] [test-akka.actor.default-dispatcher-13] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower588]: org.opendaylight.controller.cluster.common.actor.Monitor@73935a59 01:12:40,605 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Recovery completed in 1.364 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:40,606 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower588 with journal batch size 1000 01:12:40,606 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: SnapshotOffer called. 01:12:40,606 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to -1 01:12:40,606 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=1, votedFor=null 01:12:40,606 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=akka://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=akka://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower587, address=akka://test/user/follower587, votingState=VOTING]] 01:12:40,606 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower588 in 108.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:40,607 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Recovery completed in 696.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:40.706] [test-akka.actor.default-dispatcher-13] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower589]: org.opendaylight.controller.cluster.common.actor.Monitor@4941f18f 01:12:40,707 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower589 with journal batch size 1000 01:12:40,708 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: SnapshotOffer called. 01:12:40,708 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to -1 01:12:40,708 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=1, votedFor=null 01:12:40,708 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Updated server config: isVoting: false, peers: [PeerInfo [id=leader586, address=akka://test/user/leader586, votingState=VOTING], PeerInfo [id=follower588, address=akka://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=akka://test/user/follower587, votingState=VOTING]] 01:12:40,708 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower589 in 127.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [10/18/2024 01:12:40.709] [test-akka.actor.default-dispatcher-13] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader586]: org.opendaylight.controller.cluster.common.actor.Monitor@1423d22b 01:12:40,709 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader586 with journal batch size 1000 01:12:40,710 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Recovery completed in 1.490 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:40,710 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: SnapshotOffer called. 01:12:40,710 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to -1 01:12:40,710 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=1, votedFor=null 01:12:40,710 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Updated server config: isVoting: true, peers: [PeerInfo [id=follower589, address=akka://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=akka://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=akka://test/user/follower587, votingState=VOTING]] 01:12:40,710 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader586 in 99.03 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:40,711 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Recovery completed in 809.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:41,33 AM [test-akka.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:12:41,33 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.33:25520, Removed) 01:12:41,33 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Received ElectionTimeout - switching to Candidate 01:12:41,33 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Election: Candidate has following voting peers: [follower588, follower587] 01:12:41,33 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=2, votedFor=leader586 01:12:41,33 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Starting new election term 2 01:12:41,33 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] to peer follower588 01:12:41,34 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] to peer follower587 01:12:41,34 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:41,34 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:12:41,34 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 01:12:41,34 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:41,34 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:41,34 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:41,34 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=2, votedFor=null 01:12:41,34 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Cluster state: [] 01:12:41,34 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:41,34 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:41,34 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=2, votedFor=null 01:12:41,34 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:41,34 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): In requestVote: RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:41,34 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=2, votedFor=leader586 01:12:41,34 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=2, votedFor=leader586 01:12:41,34 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:41,34 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:41,35 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:12:41,36 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:12:41,36 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Election: Leader has following peers: [follower589, follower588, follower587] 01:12:41,36 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:41,36 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:41,36 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:41,36 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 2 in "AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:12:41,36 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=2, votedFor=null 01:12:41,36 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:12:41,36 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower587: Last sync leader does not match current leader leader586, need to catch up to -1 01:12:41,36 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader leader586, need to catch up to -1 01:12:41,36 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: -1, nextIndex: 0 01:12:41,36 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:41.036] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower587] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@736aaaa1 01:12:41,37 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: -1, nextIndex: 0 01:12:41,37 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:41,37 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader leader586, need to catch up to -1 01:12:41,37 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: -1, nextIndex: 0 [WARN] [10/18/2024 01:12:41.037] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower589] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@d71ca1b [WARN] [10/18/2024 01:12:41.037] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower588] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@231b7ce2 01:12:41,37 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:41,61 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:12:41,61 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 01:12:41,61 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:41,61 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:41,61 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower589 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower589: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower587 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower587: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Number of entries to be appended = 1 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Append entry to log zero 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Log size is now 1 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower589, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 01:12:41,62 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Number of entries to be appended = 1 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Append entry to log zero 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Log size is now 1 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower587, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: 0, nextIndex: 1 01:12:41,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,63 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:41,63 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:12:41,63 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Setting last applied to 0 01:12:41,63 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to 0 01:12:41,63 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Applying state for log index 0 data zero 01:12:41,64 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: applyState called: zero 01:12:41,64 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persisting ApplyJournalEntries with index=0 01:12:41,64 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:41,65 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [10/18/2024 01:12:41.065] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower589] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@bbbeefc [WARN] [10/18/2024 01:12:41.065] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower587] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@33277bb2 01:12:41,154 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:12:41,154 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:41,273 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:12:41,274 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:41,274 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Commit index set to 0 01:12:41,274 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:12:41,274 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:12:41,274 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Commit index set to 0 01:12:41,274 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Setting last applied to 0 01:12:41,274 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:12:41,274 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to 0 01:12:41,274 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:12:41,274 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Applying state for log index 0 data zero 01:12:41,274 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Setting last applied to 0 01:12:41,274 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to 0 01:12:41,274 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Applying state for log index 0 data zero 01:12:41,274 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: applyState called: zero 01:12:41,274 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Persisting ApplyJournalEntries with index=0 01:12:41,274 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: applyState called: zero 01:12:41,275 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Persisting ApplyJournalEntries with index=0 01:12:41,313 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 01:12:41,313 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 01:12:41,313 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: onShutDown 01:12:41,313 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Initiating leader transfer 01:12:41,314 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 01:12:41,314 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader586: pauseLeader successfully completed - doing transfer 01:12:41,314 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Attempting to transfer leadership 01:12:41,314 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 01:12:41,314 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,314 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,314 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:12:41,314 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,314 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:12:41,314 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower589, matchIndex: 0, lastIndex: 0, isVoting: false 01:12:41,314 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:12:41,314 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower587, matchIndex: 0, lastIndex: 0, isVoting: true 01:12:41,314 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Follower's log matches - sending ElectionTimeout 01:12:41,314 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,314 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,314 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:12:41,314 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,314 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:12:41,314 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leader transfer complete 01:12:41,314 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader transfer complete - waiting for new leader 01:12:41,314 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:12:41,315 AM [test-akka.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:12:41,315 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Received TimeoutNow - switching to Candidate 01:12:41,315 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Election: Candidate has following voting peers: [leader586, follower588] 01:12:41,315 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=3, votedFor=follower587 01:12:41,315 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Starting new election term 3 01:12:41,315 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] to peer leader586 01:12:41,315 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] to peer follower588 01:12:41,315 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:41,315 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:12:41,315 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:41,315 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Cluster state: [] 01:12:41,315 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 01:12:41,315 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2]" message is greater than leader's term 2 - switching to Follower 01:12:41,315 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=3, votedFor=null 01:12:41,315 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 01:12:41,315 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 01:12:41,315 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2]" message is greater than follower's term 2 - updating term 01:12:41,315 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leadership transfer in progress - processing RequestVote 01:12:41,315 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=3, votedFor=null 01:12:41,315 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): In requestVote: RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 01:12:41,315 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=3, votedFor=follower587 01:12:41,315 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:12:41,316 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader) :- Switching from behavior Leader to Follower, election term: 3 01:12:41,316 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:41,316 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=3, votedFor=follower587 01:12:41,316 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:12:41,321 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 01:12:41,321 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 01:12:41,321 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Election: Leader has following peers: [leader586, follower589, follower588] 01:12:41,321 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,321 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,321 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,321 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 3 in "AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 01:12:41,321 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 01:12:41,321 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=3, votedFor=null 01:12:41,322 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader follower587, need to catch up to -1 01:12:41,322 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 01:12:41,322 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - leader586: Last sync leader does not match current leader follower587, need to catch up to -1 01:12:41,322 AM [test-akka.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader changed to follower587 01:12:41,322 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,322 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader586 updated: matchIndex: 0, nextIndex: 1 01:12:41,322 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 01:12:41,322 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,322 AM [test-akka.actor.default-dispatcher-18] [INFO] RaftActorLeadershipTransferCohort - leader586: Successfully transferred leadership to follower587 in 8.486 ms [WARN] [10/18/2024 01:12:41.322] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower589] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@277e2a6d 01:12:41,323 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: leader transfer succeeded - sending PoisonPill [WARN] [10/18/2024 01:12:41.323] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower589] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@52139f4e 01:12:41,323 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - leader586: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) [WARN] [10/18/2024 01:12:41.323] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader586] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5ad736de [WARN] [10/18/2024 01:12:41.323] [test-akka.actor.default-dispatcher-16] [akka://test/user/leader586] received dead letter: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@37f4880a [WARN] [10/18/2024 01:12:41.833] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader586] received dead letter from TestActor[akka://test/user/follower587]: AppendEntries [leaderId=follower587, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:12:41,833 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader follower587, need to catch up to -1 01:12:41,833 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 01:12:41,834 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): sendAppendEntries: 0 is present for follower follower588 01:12:41,834 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:41,834 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries: AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:41,834 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Number of entries to be appended = 1 01:12:41,834 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:12:41,834 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Append entry to log zero 01:12:41,834 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:41,834 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Log size is now 1 01:12:41,835 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Commit index set to 0 01:12:41,835 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries returning : AppendEntriesReply [term=3, success=true, followerId=follower588, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:12:41,835 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 01:12:41,835 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:12:41,835 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Setting last applied to 0 01:12:41,835 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to 0 01:12:41,835 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: 0, nextIndex: 1 01:12:41,835 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Applying state for log index 0 data zero 01:12:41,835 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:41,835 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: applyState called: zero [WARN] [10/18/2024 01:12:41.834] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower588] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5a763355 01:12:41,836 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:41,836 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Persisting ApplyJournalEntries with index=0 [WARN] [10/18/2024 01:12:41.837] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower588] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7dbda8db 01:12:41,867 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 01:12:41,868 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 01:12:41,868 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower588 starting 01:12:41,868 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onShutDown 01:12:41,869 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower588 ending 01:12:41,869 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 01:12:41,869 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower587-notifier590#-252982885] 01:12:41,880 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower587-collector591#-815006898] 01:12:41,880 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower587] 01:12:41,880 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower588-notifier592#-1438372992] 01:12:41,881 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower588-collector593#236987858] 01:12:41,881 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower588] 01:12:41,881 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower589-notifier594#-18065033] 01:12:41,881 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower589-collector595#-2072512124] 01:12:41,881 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower589] 01:12:41,881 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader586-notifier596#-1601601068] 01:12:41,882 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader586-collector597#-1987977720] 01:12:41,882 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader586] 01:12:41,883 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [10/18/2024 01:12:41.885] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader598]: org.opendaylight.controller.cluster.common.actor.Monitor@3d06e42d 01:12:41,885 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader598 with journal batch size 1000 01:12:41,887 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:41,888 AM [test-akka.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:12:41,888 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower): Received TimeoutNow - switching to Candidate 01:12:41,888 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Election: Candidate has following voting peers: [] 01:12:41,888 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Set currentTerm=1, votedFor=leader598 01:12:41,888 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Starting new election term 1 01:12:41,888 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 01:12:41,889 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Received ElectionTimeout 01:12:41,889 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Leader): Election: Leader has following peers: [] 01:12:41,889 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 01:12:41,889 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader598 starting 01:12:41,889 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: onShutDown 01:12:41,890 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader598 ending 01:12:41,890 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 01:12:41,890 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader598-collector602#-360565145] 01:12:41,900 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader598] 01:12:41,901 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 01:12:41,901 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/18/2024 01:12:41.903] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower604]: org.opendaylight.controller.cluster.common.actor.Monitor@a672f2f 01:12:41,904 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower604 with journal batch size 1000 01:12:41,905 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: SnapshotOffer called. 01:12:41,905 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to -1 01:12:41,905 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=1, votedFor=null 01:12:41,905 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=akka://test/user/follower605, votingState=VOTING], PeerInfo [id=follower606, address=akka://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=akka://test/user/leader603, votingState=VOTING]] 01:12:41,905 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower604 in 128.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:41,906 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Recovery completed in 1.140 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:41.906] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower605]: org.opendaylight.controller.cluster.common.actor.Monitor@1417e616 01:12:41,907 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower605 with journal batch size 1000 01:12:41,907 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: SnapshotOffer called. 01:12:41,907 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Moving last applied index from -1 to -1 01:12:41,907 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=1, votedFor=null 01:12:41,907 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Updated server config: isVoting: true, peers: [PeerInfo [id=follower604, address=akka://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=akka://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=akka://test/user/leader603, votingState=VOTING]] 01:12:41,907 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower605 in 69.35 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:41,908 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Recovery completed in 746.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:42.008] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower606]: org.opendaylight.controller.cluster.common.actor.Monitor@6091c611 01:12:42,9 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower606 with journal batch size 1000 01:12:42,10 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: SnapshotOffer called. 01:12:42,10 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to -1 01:12:42,10 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Set currentTerm=1, votedFor=null 01:12:42,10 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Updated server config: isVoting: false, peers: [PeerInfo [id=follower605, address=akka://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=akka://test/user/follower604, votingState=VOTING], PeerInfo [id=leader603, address=akka://test/user/leader603, votingState=VOTING]] 01:12:42,10 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower606 in 182.6 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:42,11 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Recovery completed in 926.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:42.011] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader603]: org.opendaylight.controller.cluster.common.actor.Monitor@6c61b1c4 01:12:42,12 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader603 with journal batch size 1000 01:12:42,12 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: SnapshotOffer called. 01:12:42,12 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to -1 01:12:42,12 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Set currentTerm=1, votedFor=null 01:12:42,12 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=akka://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=akka://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=akka://test/user/follower606, votingState=NON_VOTING]] 01:12:42,12 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader603 in 64.36 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:42,13 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Recovery completed in 821.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:42,393 AM [test-akka.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:12:42,393 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.33:25520, Removed) 01:12:42,393 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Received ElectionTimeout - switching to Candidate 01:12:42,393 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Election: Candidate has following voting peers: [follower605, follower604] 01:12:42,393 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Set currentTerm=2, votedFor=leader603 01:12:42,393 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Starting new election term 2 01:12:42,393 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] to peer follower605 01:12:42,393 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:42,393 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] to peer follower604 01:12:42,393 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Cluster state: [] 01:12:42,393 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:42,393 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:42,394 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=2, votedFor=null 01:12:42,394 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:12:42,394 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:42,394 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): In requestVote: RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:42,394 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=2, votedFor=leader603 01:12:42,394 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Cluster state: [] 01:12:42,394 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:42,394 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:42,394 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=2, votedFor=null 01:12:42,394 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:42,394 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): In requestVote: RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:42,394 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=2, votedFor=leader603 01:12:42,394 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:42,395 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:12:42,395 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:12:42,396 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Election: Leader has following peers: [follower605, follower604, follower606] 01:12:42,396 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:42,396 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:42,396 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:42,396 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower605: Last sync leader does not match current leader leader603, need to catch up to -1 01:12:42,396 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:12:42,396 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower604: Last sync leader does not match current leader leader603, need to catch up to -1 01:12:42,396 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Term 2 in "AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:12:42,396 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Set currentTerm=2, votedFor=null 01:12:42,396 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower605 updated: matchIndex: -1, nextIndex: 0 01:12:42,396 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:42,397 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower606: Last sync leader does not match current leader leader603, need to catch up to -1 01:12:42,397 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: -1, nextIndex: 0 01:12:42,397 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:42.396] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower605] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3655da09 01:12:42,397 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: -1, nextIndex: 0 01:12:42,397 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:42.397] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower604] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@16268dca [WARN] [10/18/2024 01:12:42.397] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower606] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@365a229a 01:12:42,417 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:12:42,417 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:12:42,418 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:12:42,418 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 01:12:42,418 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:42,418 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:42,418 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 01:12:42,418 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:42,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:12:42,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:42,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:42,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower604 01:12:42,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower604: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:42,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:42,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower606 01:12:42,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower606: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:42,419 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:42,419 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Number of entries to be appended = 1 01:12:42,419 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:12:42,419 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Append entry to log zero 01:12:42,419 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:42,419 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:42,419 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Number of entries to be appended = 1 01:12:42,419 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:12:42,419 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Append entry to log zero 01:12:42,419 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:42,419 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Log size is now 1 01:12:42,419 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Log size is now 1 01:12:42,419 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower604, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:12:42,419 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower606, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:12:42,419 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 01:12:42,419 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 01:12:42,419 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: 0, nextIndex: 1 01:12:42,420 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:42,420 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: 0, nextIndex: 1 01:12:42,420 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:42,421 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:42,421 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:12:42,421 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:12:42,421 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Setting last applied to 0 01:12:42,421 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to 0 01:12:42,421 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Applying state for log index 0 data zero 01:12:42,421 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: applyState called: zero [WARN] [10/18/2024 01:12:42.421] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower606] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3a06d07d 01:12:42,421 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persisting ApplyJournalEntries with index=0 01:12:42,421 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [10/18/2024 01:12:42.421] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower604] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6eb20b4f 01:12:42,522 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:12:42,523 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:42,525 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Commit index set to 0 01:12:42,525 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Commit index set to 0 01:12:42,525 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:12:42,525 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:12:42,525 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:12:42,525 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:12:42,525 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Setting last applied to 0 01:12:42,525 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Setting last applied to 0 01:12:42,525 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to 0 01:12:42,525 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to 0 01:12:42,525 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Applying state for log index 0 data zero 01:12:42,525 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Applying state for log index 0 data zero 01:12:42,526 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: applyState called: zero 01:12:42,526 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: applyState called: zero 01:12:42,526 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Persisting ApplyJournalEntries with index=0 01:12:42,526 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Persisting ApplyJournalEntries with index=0 01:12:42,570 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 01:12:42,570 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:12:42,570 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:12:42,570 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: onRequestLeadership RequestLeadership [requestedFollowerId=follower605, replyTo=Actor[akka://test/user/$c#1747924283]] 01:12:42,570 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Initiating leader transfer 01:12:42,570 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 01:12:42,570 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 01:12:42,571 AM [test-akka.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader603: pauseLeader successfully completed - doing transfer 01:12:42,571 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 01:12:42,571 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Attempting to transfer leadership 01:12:42,571 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:42,571 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:12:42,571 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:42,571 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:12:42,571 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:12:42,571 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:12:42,643 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:12:42,643 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:12:42,764 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:12:42,764 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:12:42,883 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader603: leader transfer aborted 01:12:42,884 AM [test-akka.actor.default-dispatcher-5] [WARN] RaftActorLeadershipTransferCohort - leader603: Failed to transfer leadership in 313.5 ms 01:12:42,885 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: LeadershipTransfer request from follower605 failed 01:12:42,885 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Leadership transfer expired 01:12:42,885 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:12:42,885 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:12:42,925 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 01:12:42,926 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower604-notifier607#982106213] 01:12:42,928 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower604-collector608#-1721592851] 01:12:42,928 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower604] 01:12:42,929 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower605-notifier609#-851309719] 01:12:42,929 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower605-collector610#610619165] 01:12:42,929 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower605] 01:12:42,929 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower606-notifier611#-1449451172] 01:12:42,929 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower606-collector612#-1973291561] 01:12:42,930 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower606] 01:12:42,930 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader603-notifier613#1101666958] 01:12:42,930 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader603-collector614#357378806] 01:12:42,930 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader603] 01:12:42,930 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$c#1747924283] 01:12:42,932 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 01:12:42,932 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [10/18/2024 01:12:42.934] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower616]: org.opendaylight.controller.cluster.common.actor.Monitor@79687e11 01:12:42,934 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower616 with journal batch size 1000 01:12:42,935 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: SnapshotOffer called. 01:12:42,935 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Moving last applied index from -1 to -1 01:12:42,935 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=1, votedFor=null 01:12:42,935 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Updated server config: isVoting: true, peers: [PeerInfo [id=follower618, address=akka://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=akka://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=akka://test/user/leader615, votingState=VOTING]] 01:12:42,935 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower616 in 70.79 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:42,936 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Recovery completed in 769.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:42.936] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower617]: org.opendaylight.controller.cluster.common.actor.Monitor@71cfd8af 01:12:42,936 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower617 with journal batch size 1000 01:12:42,936 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: SnapshotOffer called. 01:12:42,936 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Moving last applied index from -1 to -1 01:12:42,936 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=1, votedFor=null 01:12:42,937 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=akka://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=akka://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=leader615, address=akka://test/user/leader615, votingState=VOTING]] 01:12:42,937 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower617 in 56.21 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:42,937 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Recovery completed in 774.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:42.938] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower618]: org.opendaylight.controller.cluster.common.actor.Monitor@6ad37abb 01:12:42,938 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower618 with journal batch size 1000 01:12:42,938 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: SnapshotOffer called. 01:12:42,938 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Moving last applied index from -1 to -1 01:12:42,938 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Set currentTerm=1, votedFor=null 01:12:42,938 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Updated server config: isVoting: false, peers: [PeerInfo [id=follower616, address=akka://test/user/follower616, votingState=VOTING], PeerInfo [id=follower617, address=akka://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=akka://test/user/leader615, votingState=VOTING]] 01:12:42,938 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower618 in 28.99 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:42,939 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Recovery completed in 460.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [10/18/2024 01:12:42.944] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader615]: org.opendaylight.controller.cluster.common.actor.Monitor@65ad35dd 01:12:42,944 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader615 with journal batch size 1000 01:12:42,944 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: SnapshotOffer called. 01:12:42,944 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Moving last applied index from -1 to -1 01:12:42,944 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=1, votedFor=null 01:12:42,944 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=akka://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=akka://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=akka://test/user/follower617, votingState=VOTING]] 01:12:42,944 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader615 in 65.93 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:12:42,945 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Recovery completed in 552.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:12:43,303 AM [test-akka.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:12:43,303 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.33:25520, Removed) 01:12:43,303 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Received ElectionTimeout - switching to Candidate 01:12:43,303 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Election: Candidate has following voting peers: [follower616, follower617] 01:12:43,303 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=2, votedFor=leader615 01:12:43,303 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Starting new election term 2 01:12:43,303 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] to peer follower616 01:12:43,303 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] to peer follower617 01:12:43,303 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:12:43,303 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:43,303 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Cluster state: [] 01:12:43,303 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:43,304 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:43,304 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=2, votedFor=null 01:12:43,304 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:43,304 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Cluster state: [] 01:12:43,304 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:43,304 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:12:43,304 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=2, votedFor=null 01:12:43,304 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): In requestVote: RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:43,304 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=2, votedFor=leader615 01:12:43,304 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:43,304 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:12:43,304 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:12:43,304 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): In requestVote: RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:43,304 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=2, votedFor=leader615 01:12:43,304 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:12:43,304 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Election: Leader has following peers: [follower616, follower618, follower617] 01:12:43,304 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,305 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,305 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,305 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:12:43,305 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618 (Follower): Term 2 in "AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:12:43,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Set currentTerm=2, votedFor=null 01:12:43,305 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower617: Last sync leader does not match current leader leader615, need to catch up to -1 01:12:43,305 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower616: Last sync leader does not match current leader leader615, need to catch up to -1 01:12:43,305 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower618: Last sync leader does not match current leader leader615, need to catch up to -1 [WARN] [10/18/2024 01:12:43.305] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower616] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@70b0ef75 [WARN] [10/18/2024 01:12:43.305] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower617] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@f5246d7 01:12:43,305 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower617 updated: matchIndex: -1, nextIndex: 0 01:12:43,305 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:43.305] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower618] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@780beece 01:12:43,305 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower616 updated: matchIndex: -1, nextIndex: 0 01:12:43,306 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,306 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower618 updated: matchIndex: -1, nextIndex: 0 01:12:43,306 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,351 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:12:43,351 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:12:43,352 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:12:43,352 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower616] 01:12:43,363 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower618] 01:12:43,374 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:12:43,374 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:12:43,374 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: onRequestLeadership RequestLeadership [requestedFollowerId=follower617, replyTo=Actor[akka://test/user/$d#-1812809981]] 01:12:43,374 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Initiating leader transfer 01:12:43,375 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader615: pauseLeader successfully completed - doing transfer 01:12:43,375 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Attempting to transfer leadership 01:12:43,375 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: onLeaderTransitioning: LeaderTransitioning [leaderId=leader615] 01:12:43,375 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,375 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:43.375] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower616] received dead letter from TestActor[akka://test/user/leader615]: LeaderTransitioning [leaderId=leader615] [WARN] [10/18/2024 01:12:43.375] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower618] received dead letter from TestActor[akka://test/user/leader615]: LeaderTransitioning [leaderId=leader615] 01:12:43,375 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:43.375] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower616] received dead letter from TestActor[akka://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [10/18/2024 01:12:43.375] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower618] received dead letter from TestActor[akka://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:12:43,375 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) [WARN] [10/18/2024 01:12:43.375] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower617] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@17264c87 01:12:43,376 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): tryToCompleteLeadershipTransfer: followerId: follower617, matchIndex: -1, lastIndex: -1, isVoting: true 01:12:43,376 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Follower's log matches - sending ElectionTimeout 01:12:43,376 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:43.376] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower616] received dead letter from TestActor[akka://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:12:43,376 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [10/18/2024 01:12:43.376] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower618] received dead letter from TestActor[akka://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:12:43,376 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,376 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 01:12:43,377 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leader transfer complete 01:12:43,377 AM [test-akka.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader transfer complete - waiting for new leader 01:12:43,377 AM [test-akka.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:12:43,377 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Received TimeoutNow - switching to Candidate 01:12:43,377 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Election: Candidate has following voting peers: [follower616, leader615] 01:12:43,377 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=3, votedFor=follower617 01:12:43,377 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Starting new election term 3 01:12:43,377 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] to peer follower616 01:12:43,377 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] to peer leader615 01:12:43,377 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:12:43,377 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:12:43,377 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Cluster state: [] 01:12:43,377 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:12:43,378 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 01:12:43,378 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=3, votedFor=null 01:12:43,378 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leadership transfer in progress - processing RequestVote 01:12:43,378 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): In requestVote: RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:12:43,378 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=3, votedFor=follower617 01:12:43,378 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:12:43,378 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader) :- Switching from behavior Leader to Follower, election term: 3 [WARN] [10/18/2024 01:12:43.378] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower616] received dead letter from TestActor[akka://test/user/follower617]: RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] 01:12:43,379 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 01:12:43,379 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 01:12:43,379 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Election: Leader has following peers: [follower616, follower618, leader615] 01:12:43,379 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,379 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,379 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,379 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 01:12:43,379 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - leader615: Last sync leader does not match current leader follower617, need to catch up to -1 01:12:43,379 AM [test-akka.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader changed to follower617 01:12:43,379 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader615 updated: matchIndex: -1, nextIndex: 0 01:12:43,380 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:12:43,379 AM [test-akka.actor.default-dispatcher-13] [INFO] RaftActorLeadershipTransferCohort - leader615: Successfully transferred leadership to follower617 in 5.090 ms 01:12:43,380 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Leadership transferred successfully to follower617 [WARN] [10/18/2024 01:12:43.379] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower616] received dead letter from TestActor[akka://test/user/follower617]: AppendEntries [leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [10/18/2024 01:12:43.380] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower618] received dead letter from TestActor[akka://test/user/follower617]: AppendEntries [leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [10/18/2024 01:12:43.380] [test-akka.actor.default-dispatcher-11] [akka://test/user/leader615] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6a05d5ac 01:12:43,425 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 01:12:43,425 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower616-notifier619#1969335236] 01:12:43,436 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower616-collector620#2012000685] 01:12:43,436 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower617-notifier621#130778890] 01:12:43,437 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower617-collector622#2022597482] 01:12:43,437 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower617] 01:12:43,437 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower618-notifier623#1240320074] 01:12:43,437 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower618-collector624#254908618] 01:12:43,437 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader615-notifier625#1741371187] 01:12:43,438 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader615-collector626#-311062957] 01:12:43,438 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader615] 01:12:43,438 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$d#-1812809981] [INFO] [10/18/2024 01:12:43.439] [main] [CoordinatedShutdown(akka://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 01:12:43,439 AM [test-akka.actor.default-dispatcher-6] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [10/18/2024 01:12:43.440] [test-akka.actor.default-dispatcher-6] [akka://test/system/cluster/core/daemon] unhandled message from Actor[akka://test/deadLetters]: Leave(akka://test@10.30.171.33:25520) [INFO] [10/18/2024 01:12:43.441] [test-akka.actor.internal-dispatcher-3] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.33:25520] - Exiting completed 01:12:43,441 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.33:25520] - Exiting completed 01:12:43,441 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.33:25520] - Shutting down... [INFO] [10/18/2024 01:12:43.441] [test-akka.actor.internal-dispatcher-3] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.33:25520] - Shutting down... [INFO] [10/18/2024 01:12:43.441] [test-akka.actor.internal-dispatcher-3] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.33:25520] - Successfully shut down 01:12:43,441 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.33:25520] - Successfully shut down [INFO] [10/18/2024 01:12:43.442] [test-akka.remote.default-remote-dispatcher-7] [akka://test@10.30.171.33:25520/system/remoting-terminator] Shutting down remote daemon. [INFO] [10/18/2024 01:12:43.442] [test-akka.remote.default-remote-dispatcher-7] [akka://test@10.30.171.33:25520/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 01:12:43,443 AM [test-akka.actor.default-dispatcher-13] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 01:12:43,443 AM [test-akka.actor.default-dispatcher-13] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [10/18/2024 01:12:43.445] [test-akka.remote.default-remote-dispatcher-7] [akka://test@10.30.171.33:25520/system/remoting-terminator] Remoting shut down. 01:12:43,445 AM [test-akka.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [DEBUG] [10/18/2024 01:12:43.445] [test-akka.actor.internal-dispatcher-10] [EventStream] shutting down: StandardOutLogger