[DEBUG] [03/27/2024 03:54:14.872] [main] [EventStream(akka://test)] logger log1-TestEventListener started 03:54:14,873 AM [test-akka.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [03/27/2024 03:54:14.873] [main] [EventStream(akka://test)] logger log2-Slf4jLogger started [DEBUG] [03/27/2024 03:54:14.873] [main] [EventStream(akka://test)] Default Loggers started [INFO] [03/27/2024 03:54:14.889] [main] [ArteryTransport(akka://test)] Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.171.192:25520] with UID [-5910850452896121926] 03:54:14,889 AM [test-akka.actor.default-dispatcher-7] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.171.192:25520] with UID [-5910850452896121926] [INFO] [03/27/2024 03:54:14.890] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.192:25520] - Starting up, Akka version [2.6.21] ... 03:54:14,892 AM [test-akka.actor.default-dispatcher-7] [INFO] Cluster - Cluster Node [akka://test@10.30.171.192:25520] - Starting up, Akka version [2.6.21] ... [INFO] [03/27/2024 03:54:14.892] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.192:25520] - Registered cluster JMX MBean [akka:type=Cluster] [INFO] [03/27/2024 03:54:14.892] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.192:25520] - Started up successfully 03:54:14,892 AM [test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://test@10.30.171.192:25520] - Registered cluster JMX MBean [akka:type=Cluster] 03:54:14,892 AM [test-akka.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [akka://test@10.30.171.192:25520] - Started up successfully 03:54:14,894 AM [test-akka.actor.default-dispatcher-7] [INFO] Cluster - Cluster Node [akka://test@10.30.171.192:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 03:54:14,894 AM [test-akka.actor.default-dispatcher-7] [INFO] Cluster - Cluster Node [akka://test@10.30.171.192:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining [INFO] [03/27/2024 03:54:14.894] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.192:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing [INFO] [03/27/2024 03:54:14.894] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.192:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 03:54:14,903 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 03:54:14,903 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/27/2024 03:54:14.906] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower646]: org.opendaylight.controller.cluster.common.actor.Monitor@6f2487af [INFO] [akkaDeadLetter][03/27/2024 03:54:14.906] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower646] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:14,907 AM [test-akka.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower646] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:14,908 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower646 with journal batch size 1000 03:54:14,909 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646: SnapshotOffer called. 03:54:14,909 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646: Moving last applied index from -1 to -1 03:54:14,909 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646: Set currentTerm=1, votedFor=null 03:54:14,909 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646: Updated server config: isVoting: true, peers: [PeerInfo [id=follower648, address=akka://test/user/follower648, votingState=NON_VOTING], PeerInfo [id=leader645, address=akka://test/user/leader645, votingState=VOTING], PeerInfo [id=follower647, address=akka://test/user/follower647, votingState=VOTING]] 03:54:14,909 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower646 in 79.32 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [03/27/2024 03:54:14.910] [test-akka.actor.default-dispatcher-14] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower647]: org.opendaylight.controller.cluster.common.actor.Monitor@1c718484 [INFO] [akkaDeadLetter][03/27/2024 03:54:14.910] [test-akka.actor.default-dispatcher-14] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower647] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:14,910 AM [test-akka.actor.default-dispatcher-7] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower647] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:14,910 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower647 with journal batch size 1000 03:54:14,911 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647: SnapshotOffer called. 03:54:14,911 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647: Moving last applied index from -1 to -1 03:54:14,911 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647: Set currentTerm=1, votedFor=null 03:54:14,911 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647: Updated server config: isVoting: true, peers: [PeerInfo [id=follower648, address=akka://test/user/follower648, votingState=NON_VOTING], PeerInfo [id=leader645, address=akka://test/user/leader645, votingState=VOTING], PeerInfo [id=follower646, address=akka://test/user/follower646, votingState=VOTING]] 03:54:14,912 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower647 in 59.69 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:14,912 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower646: Recovery completed in 3.211 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:14,913 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower647: Recovery completed in 1.989 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:15.013] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower648]: org.opendaylight.controller.cluster.common.actor.Monitor@1d31746d [INFO] [akkaDeadLetter][03/27/2024 03:54:15.013] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower648] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,13 AM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower648] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,14 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower648 with journal batch size 1000 03:54:15,14 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower648: SnapshotOffer called. 03:54:15,14 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower648: Moving last applied index from -1 to -1 03:54:15,14 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower648: Set currentTerm=1, votedFor=null 03:54:15,15 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower648: Updated server config: isVoting: false, peers: [PeerInfo [id=leader645, address=akka://test/user/leader645, votingState=VOTING], PeerInfo [id=follower647, address=akka://test/user/follower647, votingState=VOTING], PeerInfo [id=follower646, address=akka://test/user/follower646, votingState=VOTING]] 03:54:15,15 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower648 in 89.95 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:15,16 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower648: Recovery completed in 1.054 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:15.018] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader645]: org.opendaylight.controller.cluster.common.actor.Monitor@2a26142f [INFO] [akkaDeadLetter][03/27/2024 03:54:15.019] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader645] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,19 AM [test-akka.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader645] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,19 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader645 with journal batch size 1000 03:54:15,20 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645: SnapshotOffer called. 03:54:15,20 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645: Moving last applied index from -1 to -1 03:54:15,20 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645: Set currentTerm=1, votedFor=null 03:54:15,20 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645: Updated server config: isVoting: true, peers: [PeerInfo [id=follower648, address=akka://test/user/follower648, votingState=NON_VOTING], PeerInfo [id=follower647, address=akka://test/user/follower647, votingState=VOTING], PeerInfo [id=follower646, address=akka://test/user/follower646, votingState=VOTING]] 03:54:15,20 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader645 in 61.30 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:15,21 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader645: Recovery completed in 930.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:15,373 AM [test-akka.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 03:54:15,373 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.192:25520, Removed) 03:54:15,373 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Follower): Received ElectionTimeout - switching to Candidate 03:54:15,373 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Candidate): Election: Candidate has following voting peers: [follower647, follower646] 03:54:15,373 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645: Set currentTerm=2, votedFor=leader645 03:54:15,373 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Candidate): Starting new election term 2 03:54:15,373 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Candidate): Sending RequestVote [term=2, candidateId=leader645, lastLogIndex=-1, lastLogTerm=-1] to peer follower647 03:54:15,374 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Candidate): Sending RequestVote [term=2, candidateId=leader645, lastLogIndex=-1, lastLogTerm=-1] to peer follower646 03:54:15,374 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 03:54:15,374 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:15,374 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647 (Follower): Cluster state: [] 03:54:15,374 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader645, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:15,374 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower647 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader645, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:15,374 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647: Set currentTerm=2, votedFor=null 03:54:15,374 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:15,374 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646 (Follower): Cluster state: [] 03:54:15,374 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader645, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:15,374 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower646 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader645, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:15,374 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646: Set currentTerm=2, votedFor=null 03:54:15,374 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646 (Follower): In requestVote: RequestVote [term=2, candidateId=leader645, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:15,374 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647 (Follower): In requestVote: RequestVote [term=2, candidateId=leader645, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:15,374 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646: Set currentTerm=2, votedFor=leader645 03:54:15,374 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647: Set currentTerm=2, votedFor=leader645 03:54:15,374 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:15,374 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:15,375 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 03:54:15,375 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 03:54:15,375 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Election: Leader has following peers: [follower648, follower647, follower646] 03:54:15,375 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Checking sendAppendEntries for follower follower648: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:15,376 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Checking sendAppendEntries for follower follower647: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:15,376 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Checking sendAppendEntries for follower follower646: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:15,376 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower648 (Follower): Term 2 in "AppendEntries [leaderId=leader645, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 03:54:15,376 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower648: Set currentTerm=2, votedFor=null 03:54:15,376 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 03:54:15,376 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower647: Last sync leader does not match current leader leader645, need to catch up to -1 03:54:15,376 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower648: Last sync leader does not match current leader leader645, need to catch up to -1 03:54:15,376 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower647 updated: matchIndex: -1, nextIndex: 0 03:54:15,376 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower646: Last sync leader does not match current leader leader645, need to catch up to -1 [WARN] [03/27/2024 03:54:15.376] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower647] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1bc6ff09 03:54:15,376 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Checking sendAppendEntries for follower follower647: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:15,376 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower648 updated: matchIndex: -1, nextIndex: 0 03:54:15,376 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Checking sendAppendEntries for follower follower648: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:15,376 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower646 updated: matchIndex: -1, nextIndex: 0 03:54:15,376 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Checking sendAppendEntries for follower follower646: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [INFO] [akkaDeadLetter][03/27/2024 03:54:15.376] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower647] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower647] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [03/27/2024 03:54:15.377] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower648] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@23247168 [WARN] [03/27/2024 03:54:15.377] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower646] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3ca027f 03:54:15,377 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 03:54:15,377 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader645 starting 03:54:15,377 AM [test-akka.actor.default-dispatcher-12] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower647] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,377 AM [test-akka.actor.default-dispatcher-12] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower648] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,377 AM [test-akka.actor.default-dispatcher-12] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower646] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][03/27/2024 03:54:15.377] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower648] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower648] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][03/27/2024 03:54:15.377] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower646] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower646] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645: onShutDown 03:54:15,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645: Initiating leader transfer 03:54:15,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader645: pauseLeader successfully completed - doing transfer 03:54:15,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Attempting to transfer leadership 03:54:15,379 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower648: onLeaderTransitioning: LeaderTransitioning [leaderId=leader645] 03:54:15,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Checking sendAppendEntries for follower follower648: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:15,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Checking sendAppendEntries for follower follower647: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:15,379 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower647: onLeaderTransitioning: LeaderTransitioning [leaderId=leader645] 03:54:15,379 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Checking sendAppendEntries for follower follower646: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:15,380 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower646: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,380 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower647: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,380 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower648: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,380 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower646: onLeaderTransitioning: LeaderTransitioning [leaderId=leader645] [WARN] [03/27/2024 03:54:15.380] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower648] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@65c66ce [WARN] [03/27/2024 03:54:15.380] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower647] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@263e2dd [WARN] [03/27/2024 03:54:15.380] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower646] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@47616cf5 [INFO] [akkaDeadLetter][03/27/2024 03:54:15.380] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower648] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower648] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][03/27/2024 03:54:15.380] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower647] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower647] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,380 AM [test-akka.actor.default-dispatcher-7] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower648] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,380 AM [test-akka.actor.default-dispatcher-7] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower647] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][03/27/2024 03:54:15.380] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower646] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower646] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,380 AM [test-akka.actor.default-dispatcher-13] [INFO] TestActorRef - Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower646] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 03:54:15,493 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower648: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,493 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower647: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,493 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower646: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,613 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower647: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,613 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower648: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,613 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower646: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,732 AM [test-akka.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader645: leader transfer aborted 03:54:15,733 AM [test-akka.actor.default-dispatcher-13] [WARN] RaftActorLeadershipTransferCohort - leader645: Failed to transfer leadership in 353.3 ms 03:54:15,733 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645: leader transfer failed - sending PoisonPill 03:54:15,733 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader645 (Leader): Leadership transfer expired 03:54:15,733 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower648: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,733 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower646: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) 03:54:15,733 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower647: Lagging 0 entries behind leader leader645 and reached -1 (of expected -1) [WARN] [03/27/2024 03:54:15.734] [test-akka.actor.default-dispatcher-17] [akka://test/user/leader645] received dead letter from TestActor[akka://test/user/follower648]: AppendEntriesReply [term=2, success=true, followerId=follower648, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [03/27/2024 03:54:15.734] [test-akka.actor.default-dispatcher-17] [akka://test/user/leader645] received dead letter from TestActor[akka://test/user/follower646]: AppendEntriesReply [term=2, success=true, followerId=follower646, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [03/27/2024 03:54:15.734] [test-akka.actor.default-dispatcher-17] [akka://test/user/leader645] received dead letter from TestActor[akka://test/user/follower647]: AppendEntriesReply [term=2, success=true, followerId=follower647, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:54:15,736 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader645 ending 03:54:15,737 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 03:54:15,737 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower646-notifier649#-1882267006] 03:54:15,748 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower646-collector650#354046012] 03:54:15,749 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower646] 03:54:15,749 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower647-notifier651#-2101012174] 03:54:15,749 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower647-collector652#1493114333] 03:54:15,749 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower647] 03:54:15,749 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower648-notifier653#1990104592] 03:54:15,749 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower648-collector654#-1886467006] 03:54:15,749 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower648] 03:54:15,749 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader645-notifier655#824811841] 03:54:15,749 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader645-collector656#2115506774] 03:54:15,750 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader645] 03:54:15,751 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 03:54:15,751 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/27/2024 03:54:15.752] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower658]: org.opendaylight.controller.cluster.common.actor.Monitor@13e35355 03:54:15,753 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower658 with journal batch size 1000 03:54:15,754 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: SnapshotOffer called. 03:54:15,754 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Moving last applied index from -1 to -1 03:54:15,754 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Set currentTerm=1, votedFor=null 03:54:15,754 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Updated server config: isVoting: true, peers: [PeerInfo [id=follower659, address=akka://test/user/follower659, votingState=VOTING], PeerInfo [id=leader657, address=akka://test/user/leader657, votingState=VOTING], PeerInfo [id=follower660, address=akka://test/user/follower660, votingState=NON_VOTING]] 03:54:15,754 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower658 in 136.8 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:15,755 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Recovery completed in 1.250 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:15.756] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower659]: org.opendaylight.controller.cluster.common.actor.Monitor@42c1419b 03:54:15,756 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower659 with journal batch size 1000 03:54:15,757 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659: SnapshotOffer called. 03:54:15,757 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659: Moving last applied index from -1 to -1 03:54:15,757 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659: Set currentTerm=1, votedFor=null 03:54:15,757 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659: Updated server config: isVoting: true, peers: [PeerInfo [id=leader657, address=akka://test/user/leader657, votingState=VOTING], PeerInfo [id=follower658, address=akka://test/user/follower658, votingState=VOTING], PeerInfo [id=follower660, address=akka://test/user/follower660, votingState=NON_VOTING]] 03:54:15,757 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower659 in 60.89 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:15,758 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower659: Recovery completed in 841.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:15.858] [test-akka.actor.default-dispatcher-14] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower660]: org.opendaylight.controller.cluster.common.actor.Monitor@f7ad16a 03:54:15,858 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower660 with journal batch size 1000 03:54:15,859 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower660: SnapshotOffer called. 03:54:15,859 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower660: Moving last applied index from -1 to -1 03:54:15,859 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower660: Set currentTerm=1, votedFor=null 03:54:15,859 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower660: Updated server config: isVoting: false, peers: [PeerInfo [id=follower659, address=akka://test/user/follower659, votingState=VOTING], PeerInfo [id=leader657, address=akka://test/user/leader657, votingState=VOTING], PeerInfo [id=follower658, address=akka://test/user/follower658, votingState=VOTING]] 03:54:15,859 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower660 in 75.63 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:15,860 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower660: Recovery completed in 1.425 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:15.861] [test-akka.actor.default-dispatcher-14] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader657]: org.opendaylight.controller.cluster.common.actor.Monitor@35522ea5 03:54:15,861 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader657 with journal batch size 1000 03:54:15,862 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: SnapshotOffer called. 03:54:15,862 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: Moving last applied index from -1 to -1 03:54:15,862 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: Set currentTerm=1, votedFor=null 03:54:15,862 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: Updated server config: isVoting: true, peers: [PeerInfo [id=follower659, address=akka://test/user/follower659, votingState=VOTING], PeerInfo [id=follower658, address=akka://test/user/follower658, votingState=VOTING], PeerInfo [id=follower660, address=akka://test/user/follower660, votingState=NON_VOTING]] 03:54:15,862 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader657 in 41.42 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:15,863 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: Recovery completed in 706.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:16,212 AM [test-akka.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 03:54:16,212 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.192:25520, Removed) 03:54:16,212 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Follower): Received ElectionTimeout - switching to Candidate 03:54:16,212 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Candidate): Election: Candidate has following voting peers: [follower659, follower658] 03:54:16,212 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: Set currentTerm=2, votedFor=leader657 03:54:16,212 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Candidate): Starting new election term 2 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Candidate): Sending RequestVote [term=2, candidateId=leader657, lastLogIndex=-1, lastLogTerm=-1] to peer follower659 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Candidate): Sending RequestVote [term=2, candidateId=leader657, lastLogIndex=-1, lastLogTerm=-1] to peer follower658 03:54:16,213 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 03:54:16,213 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Follower): Cluster state: [] 03:54:16,213 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader657, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:16,213 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader657, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:16,213 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659: Set currentTerm=2, votedFor=null 03:54:16,213 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Follower): In requestVote: RequestVote [term=2, candidateId=leader657, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:16,213 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659: Set currentTerm=2, votedFor=leader657 03:54:16,213 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): Cluster state: [] 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader657, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader657, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Set currentTerm=2, votedFor=null 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): In requestVote: RequestVote [term=2, candidateId=leader657, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Set currentTerm=2, votedFor=leader657 03:54:16,213 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:16,215 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 03:54:16,215 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 03:54:16,215 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Election: Leader has following peers: [follower659, follower658, follower660] 03:54:16,215 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower659: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,216 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower658: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,216 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower660: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,216 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower659: Last sync leader does not match current leader leader657, need to catch up to -1 03:54:16,216 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 [WARN] [03/27/2024 03:54:16.216] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower659] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2131e975 03:54:16,216 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower658: Last sync leader does not match current leader leader657, need to catch up to -1 [WARN] [03/27/2024 03:54:16.216] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower658] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1dd4a407 03:54:16,216 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower660 (Follower): Term 2 in "AppendEntries [leaderId=leader657, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 03:54:16,216 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower659 updated: matchIndex: -1, nextIndex: 0 03:54:16,216 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower660: Set currentTerm=2, votedFor=null 03:54:16,216 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower659: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,216 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower660: Last sync leader does not match current leader leader657, need to catch up to -1 03:54:16,216 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower658 updated: matchIndex: -1, nextIndex: 0 03:54:16,216 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower658: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,216 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower660 updated: matchIndex: -1, nextIndex: 0 03:54:16,216 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower660: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,217 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 03:54:16,217 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 03:54:16,217 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending [WARN] [03/27/2024 03:54:16.217] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower660] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4bf79092 03:54:16,217 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 03:54:16,217 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 03:54:16,218 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: onRequestLeadership RequestLeadership [requestedFollowerId=follower659, replyTo=Actor[akka://test/user/$a#-1901443306]] 03:54:16,219 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: Initiating leader transfer 03:54:16,219 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader657: pauseLeader successfully completed - doing transfer 03:54:16,219 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Attempting to transfer leadership 03:54:16,219 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower659: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,219 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower658: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,219 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower660: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,219 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower659: Lagging 0 entries behind leader leader657 and reached -1 (of expected -1) 03:54:16,219 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower658: Lagging 0 entries behind leader leader657 and reached -1 (of expected -1) 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: onLeaderTransitioning: LeaderTransitioning [leaderId=leader657] 03:54:16,220 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659: onLeaderTransitioning: LeaderTransitioning [leaderId=leader657] 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower660: Lagging 0 entries behind leader leader657 and reached -1 (of expected -1) 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower660: onLeaderTransitioning: LeaderTransitioning [leaderId=leader657] 03:54:16,220 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): tryToCompleteLeadershipTransfer: followerId: follower659, matchIndex: -1, lastIndex: -1, isVoting: true 03:54:16,220 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Follower's log matches - sending ElectionTimeout [WARN] [03/27/2024 03:54:16.220] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower658] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@39aab66e 03:54:16,220 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower659: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:16.220] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower659] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7000c6a4 03:54:16,220 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower658: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:16.220] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower660] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2300485d 03:54:16,220 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Checking sendAppendEntries for follower follower660: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,220 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Leader transfer complete 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower659: Lagging 0 entries behind leader leader657 and reached -1 (of expected -1) 03:54:16,220 AM [test-akka.actor.default-dispatcher-7] [DEBUG] RaftActorLeadershipTransferCohort - leader657: leader transfer complete - waiting for new leader 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Follower): Received TimeoutNow - switching to Candidate 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Candidate): Election: Candidate has following voting peers: [leader657, follower658] 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659: Set currentTerm=3, votedFor=follower659 03:54:16,220 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower658: Lagging 0 entries behind leader leader657 and reached -1 (of expected -1) 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Candidate): Starting new election term 3 03:54:16,220 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower660: Lagging 0 entries behind leader leader657 and reached -1 (of expected -1) 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Candidate): Sending RequestVote [term=3, candidateId=follower659, lastLogIndex=-1, lastLogTerm=-1] to peer leader657 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Candidate): Sending RequestVote [term=3, candidateId=follower659, lastLogIndex=-1, lastLogTerm=-1] to peer follower658 03:54:16,220 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 03:54:16,221 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:16,221 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Cluster state: [] 03:54:16,221 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower659, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:16,221 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower659, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 03:54:16,221 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: Set currentTerm=3, votedFor=null 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): Cluster state: [] 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower659, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:16,221 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): Leadership transfer in progress - processing RequestVote 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower659, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 2 - updating term 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Set currentTerm=3, votedFor=null 03:54:16,221 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): In requestVote: RequestVote [term=3, candidateId=follower659, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:16,221 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: Set currentTerm=3, votedFor=follower659 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): In requestVote: RequestVote [term=3, candidateId=follower659, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:16,221 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658: Set currentTerm=3, votedFor=follower659 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower658 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 03:54:16,221 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader657 (Leader) :- Switching from behavior Leader to Follower, election term: 3 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Leader): Election: Leader has following peers: [leader657, follower658, follower660] 03:54:16,221 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Leader): Checking sendAppendEntries for follower leader657: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,222 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Leader): Checking sendAppendEntries for follower follower658: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,222 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Leader): Checking sendAppendEntries for follower follower660: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,222 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 03:54:16,222 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower658: Last sync leader does not match current leader follower659, need to catch up to -1 [WARN] [03/27/2024 03:54:16.222] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower658] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@ea7da7d 03:54:16,222 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower658 updated: matchIndex: -1, nextIndex: 0 03:54:16,222 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower660 (Follower): Term 3 in "AppendEntries [leaderId=follower659, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 03:54:16,222 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower660: Set currentTerm=3, votedFor=null 03:54:16,222 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - leader657: Last sync leader does not match current leader follower659, need to catch up to -1 03:54:16,222 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Leader): Checking sendAppendEntries for follower follower658: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,222 AM [test-akka.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader657: leader changed to follower659 03:54:16,222 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower660: Last sync leader does not match current leader follower659, need to catch up to -1 03:54:16,222 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader657 updated: matchIndex: -1, nextIndex: 0 03:54:16,222 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Leader): Checking sendAppendEntries for follower leader657: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,222 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower660 updated: matchIndex: -1, nextIndex: 0 03:54:16,222 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower659 (Leader): Checking sendAppendEntries for follower follower660: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,222 AM [test-akka.actor.default-dispatcher-12] [INFO] RaftActorLeadershipTransferCohort - leader657: Successfully transferred leadership to follower659 in 2.836 ms 03:54:16,222 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader657: Leadership transferred successfully to follower659 [WARN] [03/27/2024 03:54:16.223] [test-akka.actor.default-dispatcher-7] [akka://test/user/leader657] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@384fe957 [WARN] [03/27/2024 03:54:16.223] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower660] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@25b3a1c4 03:54:16,275 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 03:54:16,276 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower658-notifier661#-1052437805] 03:54:16,287 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower658-collector662#-793228245] 03:54:16,287 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower658] 03:54:16,287 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower659-notifier663#-238738131] 03:54:16,287 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower659-collector664#-647581725] 03:54:16,287 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower659] 03:54:16,288 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower660-notifier665#93523385] 03:54:16,288 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower660-collector666#-97888788] 03:54:16,288 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower660] 03:54:16,288 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader657-notifier667#-821680610] 03:54:16,288 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader657-collector668#792801253] 03:54:16,288 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader657] 03:54:16,288 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$a#-1901443306] 03:54:16,289 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 03:54:16,290 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/27/2024 03:54:16.295] [test-akka.actor.default-dispatcher-7] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower670]: org.opendaylight.controller.cluster.common.actor.Monitor@54e19d4f 03:54:16,295 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower670 with journal batch size 1000 03:54:16,297 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: SnapshotOffer called. 03:54:16,297 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Moving last applied index from -1 to -1 03:54:16,297 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Set currentTerm=1, votedFor=null 03:54:16,297 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Updated server config: isVoting: true, peers: [PeerInfo [id=leader669, address=akka://test/user/leader669, votingState=VOTING], PeerInfo [id=follower672, address=akka://test/user/follower672, votingState=NON_VOTING], PeerInfo [id=follower671, address=akka://test/user/follower671, votingState=VOTING]] 03:54:16,297 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower670 in 75.16 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:16,298 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Recovery completed in 1.140 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:16.397] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower671]: org.opendaylight.controller.cluster.common.actor.Monitor@59aa055d 03:54:16,398 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower671 with journal batch size 1000 03:54:16,398 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671: SnapshotOffer called. 03:54:16,398 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671: Moving last applied index from -1 to -1 03:54:16,398 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671: Set currentTerm=1, votedFor=null 03:54:16,398 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671: Updated server config: isVoting: true, peers: [PeerInfo [id=leader669, address=akka://test/user/leader669, votingState=VOTING], PeerInfo [id=follower670, address=akka://test/user/follower670, votingState=VOTING], PeerInfo [id=follower672, address=akka://test/user/follower672, votingState=NON_VOTING]] 03:54:16,399 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower671 in 84.79 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:16,400 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower671: Recovery completed in 1.184 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:16.499] [test-akka.actor.default-dispatcher-7] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower672]: org.opendaylight.controller.cluster.common.actor.Monitor@5f792e94 03:54:16,499 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower672 with journal batch size 1000 03:54:16,500 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower672: SnapshotOffer called. 03:54:16,500 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower672: Moving last applied index from -1 to -1 03:54:16,500 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower672: Set currentTerm=1, votedFor=null 03:54:16,500 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower672: Updated server config: isVoting: false, peers: [PeerInfo [id=leader669, address=akka://test/user/leader669, votingState=VOTING], PeerInfo [id=follower670, address=akka://test/user/follower670, votingState=VOTING], PeerInfo [id=follower671, address=akka://test/user/follower671, votingState=VOTING]] 03:54:16,500 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower672 in 92.05 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:16,501 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower672: Recovery completed in 1.096 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:16.600] [test-akka.actor.default-dispatcher-18] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader669]: org.opendaylight.controller.cluster.common.actor.Monitor@44b8e6de 03:54:16,601 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader669 with journal batch size 1000 03:54:16,602 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669: SnapshotOffer called. 03:54:16,602 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669: Moving last applied index from -1 to -1 03:54:16,602 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669: Set currentTerm=1, votedFor=null 03:54:16,602 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669: Updated server config: isVoting: true, peers: [PeerInfo [id=follower670, address=akka://test/user/follower670, votingState=VOTING], PeerInfo [id=follower672, address=akka://test/user/follower672, votingState=NON_VOTING], PeerInfo [id=follower671, address=akka://test/user/follower671, votingState=VOTING]] 03:54:16,602 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader669 in 83.84 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:16,603 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader669: Recovery completed in 1.094 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:16,942 AM [test-akka.actor.default-dispatcher-7] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 03:54:16,943 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.192:25520, Removed) 03:54:16,943 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Follower): Received ElectionTimeout - switching to Candidate 03:54:16,943 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Candidate): Election: Candidate has following voting peers: [follower670, follower671] 03:54:16,943 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669: Set currentTerm=2, votedFor=leader669 03:54:16,943 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Candidate): Starting new election term 2 03:54:16,943 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Candidate): Sending RequestVote [term=2, candidateId=leader669, lastLogIndex=-1, lastLogTerm=-1] to peer follower670 03:54:16,943 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Candidate): Sending RequestVote [term=2, candidateId=leader669, lastLogIndex=-1, lastLogTerm=-1] to peer follower671 03:54:16,943 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 03:54:16,943 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:16,943 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670 (Follower): Cluster state: [] 03:54:16,943 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader669, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:16,943 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower670 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader669, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:16,943 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Set currentTerm=2, votedFor=null 03:54:16,943 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670 (Follower): In requestVote: RequestVote [term=2, candidateId=leader669, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:16,943 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: Set currentTerm=2, votedFor=leader669 03:54:16,943 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:16,944 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671 (Follower): Cluster state: [] 03:54:16,944 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader669, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:16,944 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:16,944 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower671 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader669, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:16,944 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671: Set currentTerm=2, votedFor=null 03:54:16,944 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671 (Follower): In requestVote: RequestVote [term=2, candidateId=leader669, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:16,944 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671: Set currentTerm=2, votedFor=leader669 03:54:16,944 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:16,944 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 03:54:16,944 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Election: Leader has following peers: [follower670, follower672, follower671] 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Checking sendAppendEntries for follower follower670: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Checking sendAppendEntries for follower follower672: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Checking sendAppendEntries for follower follower671: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 03:54:16,945 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower672 (Follower): Term 2 in "AppendEntries [leaderId=leader669, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 03:54:16,945 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower672: Set currentTerm=2, votedFor=null 03:54:16,945 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower670: Last sync leader does not match current leader leader669, need to catch up to -1 03:54:16,945 AM [test-akka.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower672: Last sync leader does not match current leader leader669, need to catch up to -1 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower670 updated: matchIndex: -1, nextIndex: 0 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Checking sendAppendEntries for follower follower670: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower672 updated: matchIndex: -1, nextIndex: 0 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Checking sendAppendEntries for follower follower672: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,945 AM [test-akka.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower671: Last sync leader does not match current leader leader669, need to catch up to -1 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower671 updated: matchIndex: -1, nextIndex: 0 03:54:16,945 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Checking sendAppendEntries for follower follower671: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:16.945] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower670] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@366ba332 [WARN] [03/27/2024 03:54:16.946] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower671] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1b6a750b [WARN] [03/27/2024 03:54:16.946] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower672] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@248cf2fc 03:54:16,952 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 03:54:16,952 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 03:54:16,952 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 03:54:16,952 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower671 starting 03:54:16,953 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower671: onShutDown 03:54:16,953 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower671 ending 03:54:16,953 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 03:54:16,953 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 03:54:16,953 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669: onRequestLeadership RequestLeadership [requestedFollowerId=follower671, replyTo=Actor[akka://test/user/$b#553317073]] 03:54:16,953 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669: Initiating leader transfer 03:54:16,953 AM [test-akka.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader669: pauseLeader successfully completed - doing transfer 03:54:16,953 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Attempting to transfer leadership 03:54:16,953 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Checking sendAppendEntries for follower follower670: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,954 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Checking sendAppendEntries for follower follower672: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,954 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Checking sendAppendEntries for follower follower671: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:16,954 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower672: Lagging 0 entries behind leader leader669 and reached -1 (of expected -1) 03:54:16,954 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower672: onLeaderTransitioning: LeaderTransitioning [leaderId=leader669] [WARN] [03/27/2024 03:54:16.954] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower671] received dead letter from TestActor[akka://test/user/leader669]: LeaderTransitioning [leaderId=leader669] [WARN] [03/27/2024 03:54:16.954] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower671] received dead letter from TestActor[akka://test/user/leader669]: AppendEntries [leaderId=leader669, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [03/27/2024 03:54:16.954] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower672] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6318f82d 03:54:16,954 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower670: Lagging 0 entries behind leader leader669 and reached -1 (of expected -1) 03:54:16,954 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower670: onLeaderTransitioning: LeaderTransitioning [leaderId=leader669] [WARN] [03/27/2024 03:54:16.955] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower670] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@16f0cfff 03:54:17,63 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower670: Lagging 0 entries behind leader leader669 and reached -1 (of expected -1) 03:54:17,63 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower672: Lagging 0 entries behind leader leader669 and reached -1 (of expected -1) [WARN] [03/27/2024 03:54:17.064] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower671] received dead letter from TestActor[akka://test/user/leader669]: AppendEntries [leaderId=leader669, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 03:54:17,183 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower670: Lagging 0 entries behind leader leader669 and reached -1 (of expected -1) 03:54:17,183 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower672: Lagging 0 entries behind leader leader669 and reached -1 (of expected -1) [WARN] [03/27/2024 03:54:17.183] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower671] received dead letter from TestActor[akka://test/user/leader669]: AppendEntries [leaderId=leader669, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 03:54:17,303 AM [test-akka.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader669: leader transfer aborted 03:54:17,303 AM [test-akka.actor.default-dispatcher-11] [WARN] RaftActorLeadershipTransferCohort - leader669: Failed to transfer leadership in 349.4 ms 03:54:17,303 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669: LeadershipTransfer request from follower671 failed 03:54:17,304 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader669 (Leader): Leadership transfer expired 03:54:17,304 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower670: Lagging 0 entries behind leader leader669 and reached -1 (of expected -1) 03:54:17,304 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower672: Lagging 0 entries behind leader leader669 and reached -1 (of expected -1) [WARN] [03/27/2024 03:54:17.304] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower671] received dead letter from TestActor[akka://test/user/leader669]: AppendEntries [leaderId=leader669, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 03:54:17,307 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 03:54:17,307 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower670-notifier673#1047106821] 03:54:17,318 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower670-collector674#1049762764] 03:54:17,318 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower670] 03:54:17,319 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower671-notifier675#-1681773238] 03:54:17,319 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower671-collector676#1092793311] 03:54:17,319 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower671] 03:54:17,319 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower672-notifier677#-1403643195] 03:54:17,320 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower672-collector678#1551331185] 03:54:17,320 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower672] 03:54:17,320 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader669-notifier679#-1748593000] 03:54:17,320 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader669-collector680#-927726841] 03:54:17,321 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader669] 03:54:17,321 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$b#553317073] 03:54:17,322 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 03:54:17,322 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/27/2024 03:54:17.324] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower682]: org.opendaylight.controller.cluster.common.actor.Monitor@3ec6091c 03:54:17,324 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower682 with journal batch size 1000 03:54:17,325 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: SnapshotOffer called. 03:54:17,325 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Moving last applied index from -1 to -1 03:54:17,325 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Set currentTerm=1, votedFor=null 03:54:17,325 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Updated server config: isVoting: true, peers: [PeerInfo [id=follower684, address=akka://test/user/follower684, votingState=NON_VOTING], PeerInfo [id=leader681, address=akka://test/user/leader681, votingState=VOTING], PeerInfo [id=follower683, address=akka://test/user/follower683, votingState=VOTING]] 03:54:17,325 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower682 in 67.95 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [03/27/2024 03:54:17.325] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower683]: org.opendaylight.controller.cluster.common.actor.Monitor@5449b3e6 03:54:17,326 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower683 with journal batch size 1000 03:54:17,326 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Recovery completed in 1.025 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:17,326 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: SnapshotOffer called. 03:54:17,326 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Moving last applied index from -1 to -1 03:54:17,326 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Set currentTerm=1, votedFor=null 03:54:17,326 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Updated server config: isVoting: true, peers: [PeerInfo [id=follower684, address=akka://test/user/follower684, votingState=NON_VOTING], PeerInfo [id=leader681, address=akka://test/user/leader681, votingState=VOTING], PeerInfo [id=follower682, address=akka://test/user/follower682, votingState=VOTING]] 03:54:17,326 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower683 in 56.48 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:17,327 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Recovery completed in 838.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:17,329 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower684 with journal batch size 1000 [WARN] [03/27/2024 03:54:17.329] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower684]: org.opendaylight.controller.cluster.common.actor.Monitor@384eab67 03:54:17,329 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: SnapshotOffer called. 03:54:17,330 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: Moving last applied index from -1 to -1 03:54:17,330 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: Set currentTerm=1, votedFor=null 03:54:17,330 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: Updated server config: isVoting: false, peers: [PeerInfo [id=leader681, address=akka://test/user/leader681, votingState=VOTING], PeerInfo [id=follower683, address=akka://test/user/follower683, votingState=VOTING], PeerInfo [id=follower682, address=akka://test/user/follower682, votingState=VOTING]] 03:54:17,330 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower684 in 53.08 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:17,330 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: Recovery completed in 735.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:17.333] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader681]: org.opendaylight.controller.cluster.common.actor.Monitor@4ae81208 03:54:17,333 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader681 with journal batch size 1000 03:54:17,334 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: SnapshotOffer called. 03:54:17,334 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Moving last applied index from -1 to -1 03:54:17,334 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Set currentTerm=1, votedFor=null 03:54:17,334 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Updated server config: isVoting: true, peers: [PeerInfo [id=follower684, address=akka://test/user/follower684, votingState=NON_VOTING], PeerInfo [id=follower683, address=akka://test/user/follower683, votingState=VOTING], PeerInfo [id=follower682, address=akka://test/user/follower682, votingState=VOTING]] 03:54:17,334 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader681 in 71.94 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:17,334 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Recovery completed in 615.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:17,712 AM [test-akka.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 03:54:17,713 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.192:25520, Removed) 03:54:17,713 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Follower): Received ElectionTimeout - switching to Candidate 03:54:17,713 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Candidate): Election: Candidate has following voting peers: [follower683, follower682] 03:54:17,713 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Set currentTerm=2, votedFor=leader681 03:54:17,713 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Candidate): Starting new election term 2 03:54:17,713 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Candidate): Sending RequestVote [term=2, candidateId=leader681, lastLogIndex=-1, lastLogTerm=-1] to peer follower683 03:54:17,713 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Candidate): Sending RequestVote [term=2, candidateId=leader681, lastLogIndex=-1, lastLogTerm=-1] to peer follower682 03:54:17,713 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:17,713 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 03:54:17,713 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Cluster state: [] 03:54:17,713 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader681, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:17,713 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader681, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:17,713 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Set currentTerm=2, votedFor=null 03:54:17,714 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): In requestVote: RequestVote [term=2, candidateId=leader681, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:17,714 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Set currentTerm=2, votedFor=leader681 03:54:17,714 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:17,714 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:17,714 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Cluster state: [] 03:54:17,714 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader681, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:17,714 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader681, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:17,714 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Set currentTerm=2, votedFor=null 03:54:17,714 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): In requestVote: RequestVote [term=2, candidateId=leader681, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:17,714 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Set currentTerm=2, votedFor=leader681 03:54:17,714 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:17,715 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 03:54:17,715 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 03:54:17,715 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Election: Leader has following peers: [follower684, follower683, follower682] 03:54:17,715 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower684: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:17,715 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower683: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:17,715 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): Term 2 in "AppendEntries [leaderId=leader681, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 03:54:17,715 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower682: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:17,715 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: Set currentTerm=2, votedFor=null 03:54:17,716 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 03:54:17,716 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower684: Last sync leader does not match current leader leader681, need to catch up to -1 03:54:17,716 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower683: Last sync leader does not match current leader leader681, need to catch up to -1 03:54:17,716 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower684 updated: matchIndex: -1, nextIndex: 0 03:54:17,716 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower684: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:17,716 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower682: Last sync leader does not match current leader leader681, need to catch up to -1 [WARN] [03/27/2024 03:54:17.716] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower683] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@75843d70 03:54:17,716 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower683 updated: matchIndex: -1, nextIndex: 0 03:54:17,716 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower683: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:17,716 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower682 updated: matchIndex: -1, nextIndex: 0 [WARN] [03/27/2024 03:54:17.716] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower682] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@95b6edf 03:54:17,716 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower682: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:17.716] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower684] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@25b45a0a 03:54:17,737 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 03:54:17,737 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 03:54:17,737 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:17,737 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:17,737 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 03:54:17,737 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower684: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,737 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): sendAppendEntries: 0 is present for follower follower684 03:54:17,737 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Sending AppendEntries to follower follower684: AppendEntries [leaderId=leader681, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:17,737 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower683: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,738 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): sendAppendEntries: 0 is present for follower follower683 03:54:17,738 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Sending AppendEntries to follower follower683: AppendEntries [leaderId=leader681, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): handleAppendEntries: AppendEntries [leaderId=leader681, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:17,738 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower682: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,738 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): sendAppendEntries: 0 is present for follower follower682 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): Number of entries to be appended = 1 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 03:54:17,738 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Sending AppendEntries to follower follower682: AppendEntries [leaderId=leader681, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): Append entry to log zero 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:17,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): handleAppendEntries: AppendEntries [leaderId=leader681, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:17,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Number of entries to be appended = 1 03:54:17,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): Log size is now 1 03:54:17,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Append entry to log zero 03:54:17,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower684, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower684: Lagging 0 entries behind leader leader681 and reached -1 (of expected -1) 03:54:17,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Log size is now 1 03:54:17,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower682, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower684 updated: matchIndex: 0, nextIndex: 1 03:54:17,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower682: Lagging 0 entries behind leader leader681 and reached -1 (of expected -1) 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower684: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower682 updated: matchIndex: 0, nextIndex: 1 03:54:17,738 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower682: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:17,738 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 03:54:17,739 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Setting last applied to 0 03:54:17,739 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Moving last applied index from -1 to 0 03:54:17,739 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Applying state for log index 0 data zero 03:54:17,739 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: applyState called: zero 03:54:17,739 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:17,739 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Persisting ApplyJournalEntries with index=0 03:54:17,739 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [03/27/2024 03:54:17.739] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower684] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@693716ba [WARN] [03/27/2024 03:54:17.739] [test-akka.actor.default-dispatcher-7] [akka://test/user/follower682] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6ace668b 03:54:17,833 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): sendAppendEntries: 0 is present for follower follower683 03:54:17,833 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Sending AppendEntries to follower follower683: AppendEntries [leaderId=leader681, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:17,952 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): sendAppendEntries: 0 is present for follower follower683 03:54:17,952 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): Commit index set to 0 03:54:17,953 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Sending AppendEntries to follower follower683: AppendEntries [leaderId=leader681, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:17,953 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower684: Lagging 0 entries behind leader leader681 and reached 0 (of expected -1) 03:54:17,953 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 03:54:17,953 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): Setting last applied to 0 03:54:17,953 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: Moving last applied index from -1 to 0 03:54:17,953 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: Applying state for log index 0 data zero 03:54:17,953 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Commit index set to 0 03:54:17,953 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower682: Lagging 0 entries behind leader leader681 and reached 0 (of expected -1) 03:54:17,953 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 03:54:17,953 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Setting last applied to 0 03:54:17,953 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Moving last applied index from -1 to 0 03:54:17,953 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Applying state for log index 0 data zero 03:54:17,953 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: applyState called: zero 03:54:17,953 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: applyState called: zero 03:54:17,953 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Persisting ApplyJournalEntries with index=0 03:54:17,953 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: Persisting ApplyJournalEntries with index=0 03:54:17,989 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 03:54:17,989 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: onShutDown 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Initiating leader transfer 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader681: pauseLeader successfully completed - doing transfer 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Attempting to transfer leadership 03:54:17,990 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: onLeaderTransitioning: LeaderTransitioning [leaderId=leader681] 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower684: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,990 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: onLeaderTransitioning: LeaderTransitioning [leaderId=leader681] 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower683: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): sendAppendEntries: 0 is present for follower follower683 03:54:17,990 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower684: Lagging 0 entries behind leader leader681 and reached 0 (of expected -1) 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower682: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,990 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower682: Lagging 0 entries behind leader leader681 and reached 0 (of expected -1) 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): tryToCompleteLeadershipTransfer: followerId: follower684, matchIndex: 0, lastIndex: 0, isVoting: false 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): tryToCompleteLeadershipTransfer: followerId: follower682, matchIndex: 0, lastIndex: 0, isVoting: true 03:54:17,990 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Follower's log matches - sending ElectionTimeout 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower684: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower683: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): sendAppendEntries: 0 is present for follower follower683 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Checking sendAppendEntries for follower follower682: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Leader transfer complete 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader681: leader transfer complete - waiting for new leader 03:54:17,991 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower682: Lagging 0 entries behind leader leader681 and reached 0 (of expected -1) 03:54:17,991 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower684: Lagging 0 entries behind leader leader681 and reached 0 (of expected -1) 03:54:17,991 AM [test-akka.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 03:54:17,991 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower): Received TimeoutNow - switching to Candidate 03:54:17,991 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Candidate): Election: Candidate has following voting peers: [leader681, follower683] 03:54:17,991 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682: Set currentTerm=3, votedFor=follower682 03:54:17,991 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Candidate): Starting new election term 3 03:54:17,991 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Candidate): Sending RequestVote [term=3, candidateId=follower682, lastLogIndex=0, lastLogTerm=2] to peer leader681 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:17,991 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Candidate): Sending RequestVote [term=3, candidateId=follower682, lastLogIndex=0, lastLogTerm=2] to peer follower683 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Cluster state: [] 03:54:17,991 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower682, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 03:54:17,991 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower682, lastLogIndex=0, lastLogTerm=2]" message is greater than leader's term 2 - switching to Follower 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Set currentTerm=3, votedFor=null 03:54:17,991 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Cluster state: [] 03:54:17,991 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower682, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): Leadership transfer in progress - processing RequestVote 03:54:17,991 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower682, lastLogIndex=0, lastLogTerm=2]" message is greater than follower's term 2 - updating term 03:54:17,991 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Set currentTerm=3, votedFor=null 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): In requestVote: RequestVote [term=3, candidateId=follower682, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: Set currentTerm=3, votedFor=follower682 03:54:17,991 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 03:54:17,991 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): In requestVote: RequestVote [term=3, candidateId=follower682, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:17,992 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Set currentTerm=3, votedFor=follower682 03:54:17,992 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 03:54:17,992 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader681 (Leader) :- Switching from behavior Leader to Follower, election term: 3 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): Election: Leader has following peers: [follower684, leader681, follower683] 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): Checking sendAppendEntries for follower follower684: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): Checking sendAppendEntries for follower leader681: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,993 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower684 (Follower): Term 3 in "AppendEntries [leaderId=follower682, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): Checking sendAppendEntries for follower follower683: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,993 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower684: Set currentTerm=3, votedFor=null 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 03:54:17,993 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - leader681: Last sync leader does not match current leader follower682, need to catch up to -1 03:54:17,993 AM [test-akka.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower684: Last sync leader does not match current leader follower682, need to catch up to -1 03:54:17,993 AM [test-akka.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader681: leader changed to follower682 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader681 updated: matchIndex: 0, nextIndex: 1 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): Checking sendAppendEntries for follower leader681: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,993 AM [test-akka.actor.default-dispatcher-13] [INFO] RaftActorLeadershipTransferCohort - leader681: Successfully transferred leadership to follower682 in 3.308 ms 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower684 updated: matchIndex: 0, nextIndex: 1 03:54:17,993 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): Checking sendAppendEntries for follower follower684: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:17,994 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader681: leader transfer succeeded - sending PoisonPill 03:54:17,994 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - leader681: Lagging 0 entries behind leader follower682 and reached 0 (of expected -1) 03:54:17,994 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower684: Lagging 0 entries behind leader follower682 and reached 0 (of expected -1) [WARN] [03/27/2024 03:54:17.994] [test-akka.actor.default-dispatcher-17] [akka://test/user/leader681] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@15f52394 [WARN] [03/27/2024 03:54:17.994] [test-akka.actor.default-dispatcher-17] [akka://test/user/leader681] received dead letter: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4b015d71 [WARN] [03/27/2024 03:54:17.994] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower684] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1aad02a8 [WARN] [03/27/2024 03:54:17.994] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower684] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@63ef53a0 03:54:18,513 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower684: Lagging 0 entries behind leader follower682 and reached 0 (of expected -1) [WARN] [03/27/2024 03:54:18.513] [test-akka.actor.default-dispatcher-14] [akka://test/user/leader681] received dead letter from TestActor[akka://test/user/follower682]: AppendEntries [leaderId=follower682, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 03:54:18,513 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower683: Last sync leader does not match current leader follower682, need to catch up to -1 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): sendAppendEntries: 0 is present for follower follower683 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): Sending AppendEntries to follower follower683: AppendEntries [leaderId=follower682, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): handleAppendEntries: AppendEntries [leaderId=follower682, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] [WARN] [03/27/2024 03:54:18.514] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower683] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1d2a5279 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Number of entries to be appended = 1 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Append entry to log zero 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Log size is now 1 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Commit index set to 0 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): handleAppendEntries returning : AppendEntriesReply [term=3, success=true, followerId=follower683, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower683: Lagging 0 entries behind leader follower682 and reached 0 (of expected -1) 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683 (Follower): Setting last applied to 0 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Moving last applied index from -1 to 0 03:54:18,514 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower683 updated: matchIndex: 0, nextIndex: 1 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Applying state for log index 0 data zero 03:54:18,514 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower682 (Leader): Checking sendAppendEntries for follower follower683: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:18,514 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: applyState called: zero 03:54:18,515 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:18,515 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: Persisting ApplyJournalEntries with index=0 [WARN] [03/27/2024 03:54:18.515] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower683] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1abba581 03:54:18,545 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: onLeaderTransitioning: LeaderTransitioning [leaderId=leader681] 03:54:18,546 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 03:54:18,546 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower683 starting 03:54:18,546 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower683: onShutDown 03:54:18,546 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower683 ending 03:54:18,546 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 03:54:18,546 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower682-notifier685#-1683535198] 03:54:18,557 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower682-collector686#-1200421332] 03:54:18,558 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower682] 03:54:18,558 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower683-notifier687#-1759346874] 03:54:18,558 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower683-collector688#-65078142] 03:54:18,558 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower683] 03:54:18,558 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower684-notifier689#813401357] 03:54:18,558 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower684-collector690#-999015172] 03:54:18,559 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower684] 03:54:18,559 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader681-notifier691#1470488535] 03:54:18,559 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader681-collector692#-2023130101] 03:54:18,559 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader681] 03:54:18,560 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [03/27/2024 03:54:18.562] [test-akka.actor.default-dispatcher-14] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader693]: org.opendaylight.controller.cluster.common.actor.Monitor@7ba89f48 03:54:18,562 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader693 with journal batch size 1000 03:54:18,564 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader693: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:18,564 AM [test-akka.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 03:54:18,564 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader693 (Follower): Received TimeoutNow - switching to Candidate 03:54:18,564 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader693 (Candidate): Election: Candidate has following voting peers: [] 03:54:18,564 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader693: Set currentTerm=1, votedFor=leader693 03:54:18,564 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader693 (Candidate): Starting new election term 1 03:54:18,564 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader693 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 03:54:18,565 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader693 (Candidate): Received ElectionTimeout 03:54:18,565 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader693 (Leader): Election: Leader has following peers: [] 03:54:18,565 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader693 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 03:54:18,566 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader693 starting 03:54:18,566 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader693: onShutDown 03:54:18,566 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader693 ending 03:54:18,566 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 03:54:18,566 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader693-collector697#-1860754186] 03:54:18,577 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader693] 03:54:18,578 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 03:54:18,578 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/27/2024 03:54:18.580] [test-akka.actor.default-dispatcher-14] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower699]: org.opendaylight.controller.cluster.common.actor.Monitor@60edffa3 03:54:18,580 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower699 with journal batch size 1000 03:54:18,581 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: SnapshotOffer called. 03:54:18,581 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: Moving last applied index from -1 to -1 03:54:18,581 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: Set currentTerm=1, votedFor=null 03:54:18,581 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: Updated server config: isVoting: true, peers: [PeerInfo [id=follower700, address=akka://test/user/follower700, votingState=VOTING], PeerInfo [id=leader698, address=akka://test/user/leader698, votingState=VOTING], PeerInfo [id=follower701, address=akka://test/user/follower701, votingState=NON_VOTING]] 03:54:18,581 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower699 in 66.76 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:18,582 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: Recovery completed in 762.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:18.582] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower700]: org.opendaylight.controller.cluster.common.actor.Monitor@def0c24 03:54:18,582 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower700 with journal batch size 1000 03:54:18,583 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700: SnapshotOffer called. 03:54:18,583 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700: Moving last applied index from -1 to -1 03:54:18,583 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700: Set currentTerm=1, votedFor=null 03:54:18,583 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700: Updated server config: isVoting: true, peers: [PeerInfo [id=follower699, address=akka://test/user/follower699, votingState=VOTING], PeerInfo [id=leader698, address=akka://test/user/leader698, votingState=VOTING], PeerInfo [id=follower701, address=akka://test/user/follower701, votingState=NON_VOTING]] 03:54:18,584 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower700 in 86.99 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:18,584 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower700: Recovery completed in 784.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:18.684] [test-akka.actor.default-dispatcher-14] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower701]: org.opendaylight.controller.cluster.common.actor.Monitor@1bc437e3 03:54:18,685 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower701 with journal batch size 1000 03:54:18,685 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: SnapshotOffer called. 03:54:18,685 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: Moving last applied index from -1 to -1 03:54:18,685 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: Set currentTerm=1, votedFor=null 03:54:18,686 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: Updated server config: isVoting: false, peers: [PeerInfo [id=follower700, address=akka://test/user/follower700, votingState=VOTING], PeerInfo [id=follower699, address=akka://test/user/follower699, votingState=VOTING], PeerInfo [id=leader698, address=akka://test/user/leader698, votingState=VOTING]] 03:54:18,686 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower701 in 110.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:18,687 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: Recovery completed in 1.096 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:18.688] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader698]: org.opendaylight.controller.cluster.common.actor.Monitor@216bd295 03:54:18,688 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader698 with journal batch size 1000 03:54:18,689 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: SnapshotOffer called. 03:54:18,689 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Moving last applied index from -1 to -1 03:54:18,689 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Set currentTerm=1, votedFor=null 03:54:18,689 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Updated server config: isVoting: true, peers: [PeerInfo [id=follower700, address=akka://test/user/follower700, votingState=VOTING], PeerInfo [id=follower699, address=akka://test/user/follower699, votingState=VOTING], PeerInfo [id=follower701, address=akka://test/user/follower701, votingState=NON_VOTING]] 03:54:18,689 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader698 in 68.89 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:18,690 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Recovery completed in 689.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:19,22 AM [test-akka.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 03:54:19,22 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.192:25520, Removed) 03:54:19,22 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Follower): Received ElectionTimeout - switching to Candidate 03:54:19,22 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Candidate): Election: Candidate has following voting peers: [follower700, follower699] 03:54:19,22 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Set currentTerm=2, votedFor=leader698 03:54:19,22 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Candidate): Starting new election term 2 03:54:19,23 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Candidate): Sending RequestVote [term=2, candidateId=leader698, lastLogIndex=-1, lastLogTerm=-1] to peer follower700 03:54:19,23 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Candidate): Sending RequestVote [term=2, candidateId=leader698, lastLogIndex=-1, lastLogTerm=-1] to peer follower699 03:54:19,23 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:19,23 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 03:54:19,23 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700 (Follower): Cluster state: [] 03:54:19,23 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader698, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:19,23 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:19,23 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower700 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader698, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:19,23 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700: Set currentTerm=2, votedFor=null 03:54:19,23 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): Cluster state: [] 03:54:19,23 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700 (Follower): In requestVote: RequestVote [term=2, candidateId=leader698, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:19,23 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader698, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:19,23 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700: Set currentTerm=2, votedFor=leader698 03:54:19,23 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader698, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:19,23 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: Set currentTerm=2, votedFor=null 03:54:19,23 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:19,23 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): In requestVote: RequestVote [term=2, candidateId=leader698, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:19,23 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: Set currentTerm=2, votedFor=leader698 03:54:19,23 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Election: Leader has following peers: [follower700, follower699, follower701] 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower700: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower699: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower701: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:19,24 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower700: Last sync leader does not match current leader leader698, need to catch up to -1 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower700 updated: matchIndex: -1, nextIndex: 0 03:54:19,24 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower699: Last sync leader does not match current leader leader698, need to catch up to -1 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower700: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:19.024] [test-akka.actor.default-dispatcher-13] [akka://test/user/follower700] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@210f41a4 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower701 (Follower): Term 2 in "AppendEntries [leaderId=leader698, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 03:54:19,24 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: Set currentTerm=2, votedFor=null 03:54:19,25 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower699 updated: matchIndex: -1, nextIndex: 0 03:54:19,25 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower701: Last sync leader does not match current leader leader698, need to catch up to -1 [WARN] [03/27/2024 03:54:19.025] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower699] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1aea791c 03:54:19,25 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower699: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:19,25 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower701 updated: matchIndex: -1, nextIndex: 0 03:54:19,25 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower701: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:19.025] [test-akka.actor.default-dispatcher-7] [akka://test/user/follower701] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@26b0d915 03:54:19,48 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 03:54:19,48 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 03:54:19,48 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 03:54:19,48 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 03:54:19,48 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:19,48 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower700: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): sendAppendEntries: 0 is present for follower follower700 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Sending AppendEntries to follower follower700: AppendEntries [leaderId=leader698, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower699: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): sendAppendEntries: 0 is present for follower follower699 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Sending AppendEntries to follower follower699: AppendEntries [leaderId=leader698, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower701: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): sendAppendEntries: 0 is present for follower follower701 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Sending AppendEntries to follower follower701: AppendEntries [leaderId=leader698, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): handleAppendEntries: AppendEntries [leaderId=leader698, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): Number of entries to be appended = 1 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): Append entry to log zero 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): Log size is now 1 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower699, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower699: Lagging 0 entries behind leader leader698 and reached -1 (of expected -1) 03:54:19,49 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701 (Follower): handleAppendEntries: AppendEntries [leaderId=leader698, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:19,49 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701 (Follower): Number of entries to be appended = 1 03:54:19,49 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower699 updated: matchIndex: 0, nextIndex: 1 03:54:19,49 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701 (Follower): Append entry to log zero 03:54:19,49 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:19,49 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower699: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:19,49 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701 (Follower): Log size is now 1 03:54:19,49 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower701, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:54:19,49 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower701: Lagging 0 entries behind leader leader698 and reached -1 (of expected -1) 03:54:19,50 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower701 updated: matchIndex: 0, nextIndex: 1 03:54:19,50 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower701: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:19,50 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:19,50 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 03:54:19,50 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Setting last applied to 0 03:54:19,50 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Moving last applied index from -1 to 0 03:54:19,50 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Applying state for log index 0 data zero 03:54:19,50 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: applyState called: zero 03:54:19,51 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Persisting ApplyJournalEntries with index=0 03:54:19,51 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 03:54:19,51 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [03/27/2024 03:54:19.051] [test-akka.actor.default-dispatcher-14] [akka://test/user/follower701] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7e59f159 [WARN] [03/27/2024 03:54:19.051] [test-akka.actor.default-dispatcher-14] [akka://test/user/follower699] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@78f476f5 03:54:19,143 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): sendAppendEntries: 0 is present for follower follower700 03:54:19,143 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Sending AppendEntries to follower follower700: AppendEntries [leaderId=leader698, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:19,263 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): sendAppendEntries: 0 is present for follower follower700 03:54:19,263 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Sending AppendEntries to follower follower700: AppendEntries [leaderId=leader698, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 03:54:19,263 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): Commit index set to 0 03:54:19,263 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701 (Follower): Commit index set to 0 03:54:19,263 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower699: Lagging 0 entries behind leader leader698 and reached 0 (of expected -1) 03:54:19,263 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower701: Lagging 0 entries behind leader leader698 and reached 0 (of expected -1) 03:54:19,263 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 03:54:19,263 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 03:54:19,263 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701 (Follower): Setting last applied to 0 03:54:19,263 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699 (Follower): Setting last applied to 0 03:54:19,263 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: Moving last applied index from -1 to 0 03:54:19,263 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: Moving last applied index from -1 to 0 03:54:19,263 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: Applying state for log index 0 data zero 03:54:19,263 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: Applying state for log index 0 data zero 03:54:19,263 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: applyState called: zero 03:54:19,263 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: applyState called: zero 03:54:19,264 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: Persisting ApplyJournalEntries with index=0 03:54:19,264 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: Persisting ApplyJournalEntries with index=0 03:54:19,301 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 03:54:19,301 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 03:54:19,301 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 03:54:19,301 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: onRequestLeadership RequestLeadership [requestedFollowerId=follower700, replyTo=Actor[akka://test/user/$c#-673747551]] 03:54:19,302 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: Initiating leader transfer 03:54:19,302 AM [test-akka.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader698: pauseLeader successfully completed - doing transfer 03:54:19,302 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Attempting to transfer leadership 03:54:19,302 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower700: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:19,302 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower699: onLeaderTransitioning: LeaderTransitioning [leaderId=leader698] 03:54:19,302 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): sendAppendEntries: 0 is present for follower follower700 03:54:19,302 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower700: onLeaderTransitioning: LeaderTransitioning [leaderId=leader698] 03:54:19,302 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower699: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:19,302 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower701: onLeaderTransitioning: LeaderTransitioning [leaderId=leader698] 03:54:19,302 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Checking sendAppendEntries for follower follower701: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 03:54:19,302 AM [test-akka.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower699: Lagging 0 entries behind leader leader698 and reached 0 (of expected -1) 03:54:19,302 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower701: Lagging 0 entries behind leader leader698 and reached 0 (of expected -1) 03:54:19,503 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower699: Lagging 0 entries behind leader leader698 and reached 0 (of expected -1) 03:54:19,503 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower701: Lagging 0 entries behind leader leader698 and reached 0 (of expected -1) 03:54:19,622 AM [test-akka.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader698: leader transfer aborted 03:54:19,622 AM [test-akka.actor.default-dispatcher-6] [WARN] RaftActorLeadershipTransferCohort - leader698: Failed to transfer leadership in 320.4 ms 03:54:19,622 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698: LeadershipTransfer request from follower700 failed 03:54:19,623 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader698 (Leader): Leadership transfer expired 03:54:19,623 AM [test-akka.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower699: Lagging 0 entries behind leader leader698 and reached 0 (of expected -1) 03:54:19,623 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower701: Lagging 0 entries behind leader leader698 and reached 0 (of expected -1) 03:54:19,655 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 03:54:19,655 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower699-notifier702#-221694257] 03:54:19,667 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower699-collector703#-1663377433] 03:54:19,667 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower699] 03:54:19,668 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower700-notifier704#-579051058] 03:54:19,669 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower700-collector705#-197534768] 03:54:19,671 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower700] 03:54:19,671 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower701-notifier706#-1454836930] 03:54:19,672 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower701-collector707#1030430265] 03:54:19,672 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower701] 03:54:19,672 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader698-notifier708#-753424368] 03:54:19,672 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader698-collector709#-1852802264] 03:54:19,672 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader698] 03:54:19,672 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$c#-673747551] 03:54:19,673 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 03:54:19,673 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [03/27/2024 03:54:19.678] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower711]: org.opendaylight.controller.cluster.common.actor.Monitor@69d11cf1 03:54:19,679 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower711 with journal batch size 1000 03:54:19,679 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: SnapshotOffer called. 03:54:19,679 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Moving last applied index from -1 to -1 03:54:19,679 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Set currentTerm=1, votedFor=null 03:54:19,679 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Updated server config: isVoting: true, peers: [PeerInfo [id=follower713, address=akka://test/user/follower713, votingState=NON_VOTING], PeerInfo [id=leader710, address=akka://test/user/leader710, votingState=VOTING], PeerInfo [id=follower712, address=akka://test/user/follower712, votingState=VOTING]] 03:54:19,679 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower711 in 47.61 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [03/27/2024 03:54:19.680] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower712]: org.opendaylight.controller.cluster.common.actor.Monitor@d02bfd1 03:54:19,680 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower712 with journal batch size 1000 03:54:19,680 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712: SnapshotOffer called. 03:54:19,680 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Recovery completed in 824.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:19,680 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712: Moving last applied index from -1 to -1 03:54:19,680 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712: Set currentTerm=1, votedFor=null 03:54:19,680 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712: Updated server config: isVoting: true, peers: [PeerInfo [id=follower711, address=akka://test/user/follower711, votingState=VOTING], PeerInfo [id=follower713, address=akka://test/user/follower713, votingState=NON_VOTING], PeerInfo [id=leader710, address=akka://test/user/leader710, votingState=VOTING]] 03:54:19,680 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower712 in 28.92 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:19,681 AM [test-akka.actor.default-dispatcher-7] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower712: Recovery completed in 353.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [03/27/2024 03:54:19.681] [test-akka.actor.default-dispatcher-12] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower713]: org.opendaylight.controller.cluster.common.actor.Monitor@66870e98 03:54:19,681 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower713 with journal batch size 1000 [WARN] [03/27/2024 03:54:19.682] [test-akka.actor.default-dispatcher-14] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader710]: org.opendaylight.controller.cluster.common.actor.Monitor@168176a2 03:54:19,682 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader710 with journal batch size 1000 03:54:19,686 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower713: SnapshotOffer called. 03:54:19,686 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower713: Moving last applied index from -1 to -1 03:54:19,686 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower713: Set currentTerm=1, votedFor=null 03:54:19,686 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower713: Updated server config: isVoting: false, peers: [PeerInfo [id=follower711, address=akka://test/user/follower711, votingState=VOTING], PeerInfo [id=leader710, address=akka://test/user/leader710, votingState=VOTING], PeerInfo [id=follower712, address=akka://test/user/follower712, votingState=VOTING]] 03:54:19,687 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower713 in 59.16 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:19,687 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: SnapshotOffer called. 03:54:19,687 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: Moving last applied index from -1 to -1 03:54:19,687 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: Set currentTerm=1, votedFor=null 03:54:19,687 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: Updated server config: isVoting: true, peers: [PeerInfo [id=follower711, address=akka://test/user/follower711, votingState=VOTING], PeerInfo [id=follower713, address=akka://test/user/follower713, votingState=NON_VOTING], PeerInfo [id=follower712, address=akka://test/user/follower712, votingState=VOTING]] 03:54:19,687 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader710 in 33.14 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 03:54:19,690 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: Recovery completed in 3.228 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:19,690 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower713: Recovery completed in 3.332 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 03:54:20,103 AM [test-akka.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 03:54:20,103 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.192:25520, Removed) 03:54:20,103 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Follower): Received ElectionTimeout - switching to Candidate 03:54:20,103 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Candidate): Election: Candidate has following voting peers: [follower711, follower712] 03:54:20,103 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: Set currentTerm=2, votedFor=leader710 03:54:20,103 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Candidate): Starting new election term 2 03:54:20,103 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Candidate): Sending RequestVote [term=2, candidateId=leader710, lastLogIndex=-1, lastLogTerm=-1] to peer follower711 03:54:20,103 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Candidate): Sending RequestVote [term=2, candidateId=leader710, lastLogIndex=-1, lastLogTerm=-1] to peer follower712 03:54:20,104 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 03:54:20,104 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:20,104 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): Cluster state: [] 03:54:20,104 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader710, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:20,104 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader710, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:20,104 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Set currentTerm=2, votedFor=null 03:54:20,104 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:20,104 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): In requestVote: RequestVote [term=2, candidateId=leader710, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:20,104 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Follower): Cluster state: [] 03:54:20,104 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader710, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:20,104 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader710, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 03:54:20,104 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711: Set currentTerm=2, votedFor=leader710 03:54:20,104 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712: Set currentTerm=2, votedFor=null 03:54:20,104 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Follower): In requestVote: RequestVote [term=2, candidateId=leader710, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:20,104 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712: Set currentTerm=2, votedFor=leader710 03:54:20,104 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower711 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:20,104 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 03:54:20,105 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 03:54:20,105 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 03:54:20,105 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Election: Leader has following peers: [follower711, follower713, follower712] 03:54:20,105 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower711: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:20,105 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower713: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:20,105 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower712: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:20,105 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower711: Last sync leader does not match current leader leader710, need to catch up to -1 03:54:20,105 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 03:54:20,105 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower713 (Follower): Term 2 in "AppendEntries [leaderId=leader710, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 03:54:20,105 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower713: Set currentTerm=2, votedFor=null 03:54:20,105 AM [test-akka.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower713: Last sync leader does not match current leader leader710, need to catch up to -1 03:54:20,105 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower711 updated: matchIndex: -1, nextIndex: 0 [WARN] [03/27/2024 03:54:20.105] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower711] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3506e283 03:54:20,106 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower711: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:20,106 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower712: Last sync leader does not match current leader leader710, need to catch up to -1 03:54:20,106 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower713 updated: matchIndex: -1, nextIndex: 0 03:54:20,106 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower713: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:20.106] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower712] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2d7bee4d 03:54:20,106 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower712 updated: matchIndex: -1, nextIndex: 0 03:54:20,106 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower712: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:20.106] [test-akka.actor.default-dispatcher-7] [akka://test/user/follower713] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@579d6aca 03:54:20,144 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 03:54:20,144 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 03:54:20,145 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 03:54:20,145 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower711] 03:54:20,156 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower713] 03:54:20,167 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 03:54:20,167 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 03:54:20,167 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: onRequestLeadership RequestLeadership [requestedFollowerId=follower712, replyTo=Actor[akka://test/user/$d#807817369]] 03:54:20,167 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: Initiating leader transfer 03:54:20,168 AM [test-akka.actor.default-dispatcher-7] [DEBUG] RaftActorLeadershipTransferCohort - leader710: pauseLeader successfully completed - doing transfer 03:54:20,168 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712: onLeaderTransitioning: LeaderTransitioning [leaderId=leader710] 03:54:20,168 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Attempting to transfer leadership [WARN] [03/27/2024 03:54:20.168] [test-akka.actor.default-dispatcher-14] [akka://test/user/follower711] received dead letter from TestActor[akka://test/user/leader710]: LeaderTransitioning [leaderId=leader710] 03:54:20,168 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower711: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:20.168] [test-akka.actor.default-dispatcher-14] [akka://test/user/follower713] received dead letter from TestActor[akka://test/user/leader710]: LeaderTransitioning [leaderId=leader710] 03:54:20,168 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower713: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:20.168] [test-akka.actor.default-dispatcher-14] [akka://test/user/follower711] received dead letter from TestActor[akka://test/user/leader710]: AppendEntries [leaderId=leader710, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 03:54:20,168 AM [test-akka.actor.default-dispatcher-7] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower712: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:20.168] [test-akka.actor.default-dispatcher-14] [akka://test/user/follower713] received dead letter from TestActor[akka://test/user/leader710]: AppendEntries [leaderId=leader710, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 03:54:20,168 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower712: Lagging 0 entries behind leader leader710 and reached -1 (of expected -1) 03:54:20,168 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): tryToCompleteLeadershipTransfer: followerId: follower712, matchIndex: -1, lastIndex: -1, isVoting: true 03:54:20,168 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Follower's log matches - sending ElectionTimeout 03:54:20,168 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower711: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:20.168] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower712] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6a287178 03:54:20,168 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower713: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:20.168] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower711] received dead letter from TestActor[akka://test/user/leader710]: AppendEntries [leaderId=leader710, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 03:54:20,168 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Checking sendAppendEntries for follower follower712: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:20.168] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower713] received dead letter from TestActor[akka://test/user/leader710]: AppendEntries [leaderId=leader710, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 03:54:20,168 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Leader transfer complete 03:54:20,168 AM [test-akka.actor.default-dispatcher-14] [DEBUG] RaftActorLeadershipTransferCohort - leader710: leader transfer complete - waiting for new leader 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower712: Lagging 0 entries behind leader leader710 and reached -1 (of expected -1) 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Follower): Received TimeoutNow - switching to Candidate 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Candidate): Election: Candidate has following voting peers: [follower711, leader710] 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712: Set currentTerm=3, votedFor=follower712 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Candidate): Starting new election term 3 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Candidate): Sending RequestVote [term=3, candidateId=follower712, lastLogIndex=-1, lastLogTerm=-1] to peer follower711 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Candidate): Sending RequestVote [term=3, candidateId=follower712, lastLogIndex=-1, lastLogTerm=-1] to peer leader710 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 [WARN] [03/27/2024 03:54:20.169] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower711] received dead letter from TestActor[akka://test/user/follower712]: RequestVote [term=3, candidateId=follower712, lastLogIndex=-1, lastLogTerm=-1] 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Cluster state: [] 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower712, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower712, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 03:54:20,169 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: Set currentTerm=3, votedFor=null 03:54:20,170 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): Leadership transfer in progress - processing RequestVote 03:54:20,170 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): In requestVote: RequestVote [term=3, candidateId=follower712, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 03:54:20,170 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: Set currentTerm=3, votedFor=follower712 03:54:20,170 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 03:54:20,170 AM [test-akka.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader710 (Leader) :- Switching from behavior Leader to Follower, election term: 3 03:54:20,170 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 03:54:20,170 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 03:54:20,170 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Leader): Election: Leader has following peers: [follower711, follower713, leader710] 03:54:20,170 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Leader): Checking sendAppendEntries for follower follower711: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:20,170 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Leader): Checking sendAppendEntries for follower follower713: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:20,170 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Leader): Checking sendAppendEntries for follower leader710: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/27/2024 03:54:20.170] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower711] received dead letter from TestActor[akka://test/user/follower712]: AppendEntries [leaderId=follower712, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 03:54:20,170 AM [test-akka.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [03/27/2024 03:54:20.170] [test-akka.actor.default-dispatcher-12] [akka://test/user/follower713] received dead letter from TestActor[akka://test/user/follower712]: AppendEntries [leaderId=follower712, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 03:54:20,171 AM [test-akka.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - leader710: Last sync leader does not match current leader follower712, need to catch up to -1 03:54:20,171 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader710 updated: matchIndex: -1, nextIndex: 0 03:54:20,171 AM [test-akka.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader710: leader changed to follower712 03:54:20,171 AM [test-akka.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower712 (Leader): Checking sendAppendEntries for follower leader710: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:54:20,171 AM [test-akka.actor.default-dispatcher-12] [INFO] RaftActorLeadershipTransferCohort - leader710: Successfully transferred leadership to follower712 in 3.358 ms 03:54:20,171 AM [test-akka.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader710: Leadership transferred successfully to follower712 [WARN] [03/27/2024 03:54:20.171] [test-akka.actor.default-dispatcher-12] [akka://test/user/leader710] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@37de7c17 03:54:20,219 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 03:54:20,219 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower711-notifier714#1326347733] 03:54:20,230 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower711-collector715#156642393] 03:54:20,230 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower712-notifier716#-2011495415] 03:54:20,230 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower712-collector717#-817585847] 03:54:20,230 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower712] 03:54:20,231 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower713-notifier718#478050813] 03:54:20,231 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower713-collector719#231343087] 03:54:20,231 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader710-notifier720#-679097024] 03:54:20,231 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader710-collector721#-1404997648] 03:54:20,231 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader710] 03:54:20,231 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$d#807817369] [INFO] [03/27/2024 03:54:20.232] [main] [CoordinatedShutdown(akka://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 03:54:20,232 AM [test-akka.actor.default-dispatcher-14] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [03/27/2024 03:54:20.233] [test-akka.actor.default-dispatcher-17] [akka://test/system/cluster/core/daemon] unhandled message from Actor[akka://test/deadLetters]: Leave(akka://test@10.30.171.192:25520) [INFO] [03/27/2024 03:54:20.234] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.192:25520] - Exiting completed 03:54:20,234 AM [test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://test@10.30.171.192:25520] - Exiting completed 03:54:20,234 AM [test-akka.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [akka://test@10.30.171.192:25520] - Shutting down... [INFO] [03/27/2024 03:54:20.234] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.192:25520] - Shutting down... [INFO] [03/27/2024 03:54:20.234] [test-akka.actor.internal-dispatcher-4] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.192:25520] - Successfully shut down 03:54:20,234 AM [test-akka.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [akka://test@10.30.171.192:25520] - Successfully shut down [INFO] [03/27/2024 03:54:20.236] [test-akka.remote.default-remote-dispatcher-8] [akka://test@10.30.171.192:25520/system/remoting-terminator] Shutting down remote daemon. 03:54:20,236 AM [test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 03:54:20,236 AM [test-akka.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [03/27/2024 03:54:20.236] [test-akka.remote.default-remote-dispatcher-8] [akka://test@10.30.171.192:25520/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. [INFO] [03/27/2024 03:54:20.241] [test-akka.remote.default-remote-dispatcher-8] [akka://test@10.30.171.192:25520/system/remoting-terminator] Remoting shut down. 03:54:20,241 AM [test-akka.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [DEBUG] [03/27/2024 03:54:20.241] [test-akka.actor.internal-dispatcher-3] [EventStream] shutting down: StandardOutLogger