[DEBUG] [08/19/2024 01:11:20.464] [main] [EventStream(akka://test)] logger log1-TestEventListener started 01:11:20,464 AM [test-akka.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [08/19/2024 01:11:20.464] [main] [EventStream(akka://test)] logger log2-Slf4jLogger started [DEBUG] [08/19/2024 01:11:20.464] [main] [EventStream(akka://test)] Default Loggers started [INFO] [08/19/2024 01:11:20.473] [main] [ArteryTransport(akka://test)] Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.171.178:25520] with UID [226858743206122985] [INFO] [08/19/2024 01:11:20.474] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.178:25520] - Starting up, Akka version [2.6.21] ... 01:11:20,474 AM [test-akka.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [akka://test@10.30.171.178:25520] with UID [226858743206122985] 01:11:20,474 AM [test-akka.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [akka://test@10.30.171.178:25520] - Starting up, Akka version [2.6.21] ... [INFO] [08/19/2024 01:11:20.477] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.178:25520] - Registered cluster JMX MBean [akka:type=Cluster] 01:11:20,477 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@10.30.171.178:25520] - Registered cluster JMX MBean [akka:type=Cluster] [INFO] [08/19/2024 01:11:20.477] [main] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.178:25520] - Started up successfully 01:11:20,477 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@10.30.171.178:25520] - Started up successfully [INFO] [08/19/2024 01:11:20.477] [test-akka.actor.internal-dispatcher-2] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.178:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing [INFO] [08/19/2024 01:11:20.478] [test-akka.actor.internal-dispatcher-2] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.178:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 01:11:20,478 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@10.30.171.178:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing 01:11:20,478 AM [test-akka.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [akka://test@10.30.171.178:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining 01:11:20,488 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 01:11:20,488 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/19/2024 01:11:20.491] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower551]: org.opendaylight.controller.cluster.common.actor.Monitor@22edce56 [INFO] [akkaDeadLetter][08/19/2024 01:11:20.491] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower551] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,491 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower551] to Actor[akka://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,493 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower551 with journal batch size 1000 01:11:20,494 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower552 with journal batch size 1000 [WARN] [08/19/2024 01:11:20.494] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower552]: org.opendaylight.controller.cluster.common.actor.Monitor@6a73f014 [INFO] [akkaDeadLetter][08/19/2024 01:11:20.495] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower552] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,495 AM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower552] to Actor[akka://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,496 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower553 with journal batch size 1000 [WARN] [08/19/2024 01:11:20.496] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower553]: org.opendaylight.controller.cluster.common.actor.Monitor@40ef655 [INFO] [akkaDeadLetter][08/19/2024 01:11:20.496] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower553] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,496 AM [test-akka.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/follower553] to Actor[akka://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,497 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader550 with journal batch size 1000 [WARN] [08/19/2024 01:11:20.497] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader550]: org.opendaylight.controller.cluster.common.actor.Monitor@4ab930fb [INFO] [akkaDeadLetter][08/19/2024 01:11:20.497] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader550] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,497 AM [test-akka.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[akka://test/user/leader550] to Actor[akka://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,498 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: SnapshotOffer called. 01:11:20,498 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Moving last applied index from -1 to -1 01:11:20,498 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=1, votedFor=null 01:11:20,498 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=akka://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=akka://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=akka://test/user/leader550, votingState=VOTING]] 01:11:20,498 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower551 in 71.15 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:20,499 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Recovery completed in 697.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:20,499 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: SnapshotOffer called. 01:11:20,500 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Moving last applied index from -1 to -1 01:11:20,500 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=1, votedFor=null 01:11:20,500 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=akka://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=leader550, address=akka://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=akka://test/user/follower551, votingState=VOTING]] 01:11:20,500 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower552 in 71.01 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:20,500 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Recovery completed in 817.6 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:20,501 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: SnapshotOffer called. 01:11:20,501 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Moving last applied index from -1 to -1 01:11:20,501 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Set currentTerm=1, votedFor=null 01:11:20,501 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Updated server config: isVoting: false, peers: [PeerInfo [id=follower552, address=akka://test/user/follower552, votingState=VOTING], PeerInfo [id=leader550, address=akka://test/user/leader550, votingState=VOTING], PeerInfo [id=follower551, address=akka://test/user/follower551, votingState=VOTING]] 01:11:20,501 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower553 in 63.32 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:20,502 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: SnapshotOffer called. 01:11:20,502 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Moving last applied index from -1 to -1 01:11:20,502 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Set currentTerm=1, votedFor=null 01:11:20,502 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Updated server config: isVoting: true, peers: [PeerInfo [id=follower553, address=akka://test/user/follower553, votingState=NON_VOTING], PeerInfo [id=follower552, address=akka://test/user/follower552, votingState=VOTING], PeerInfo [id=follower551, address=akka://test/user/follower551, votingState=VOTING]] 01:11:20,502 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader550 in 78.16 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:20,503 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Recovery completed in 1.663 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:20,503 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Recovery completed in 567.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:20,922 AM [test-akka.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:11:20,922 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.178:25520, Removed) 01:11:20,922 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower): Received ElectionTimeout - switching to Candidate 01:11:20,922 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Election: Candidate has following voting peers: [follower552, follower551] 01:11:20,922 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Set currentTerm=2, votedFor=leader550 01:11:20,923 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Starting new election term 2 01:11:20,923 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] to peer follower552 01:11:20,923 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): Sending RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] to peer follower551 01:11:20,923 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:20,923 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:11:20,923 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Cluster state: [] 01:11:20,923 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:20,923 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:20,923 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:20,923 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Cluster state: [] 01:11:20,923 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=2, votedFor=null 01:11:20,923 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): In requestVote: RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:20,923 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Set currentTerm=2, votedFor=leader550 01:11:20,923 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:20,923 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:20,923 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:20,924 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=2, votedFor=null 01:11:20,924 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): In requestVote: RequestVote [term=2, candidateId=leader550, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:20,924 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Set currentTerm=2, votedFor=leader550 01:11:20,925 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:20,927 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:11:20,927 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:11:20,927 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Election: Leader has following peers: [follower553, follower552, follower551] 01:11:20,927 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:20,928 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower553 (Follower): Term 2 in "AppendEntries [leaderId=leader550, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:11:20,928 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: Set currentTerm=2, votedFor=null 01:11:20,928 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower553: Last sync leader does not match current leader leader550, need to catch up to -1 01:11:20,928 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:20,928 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:20,928 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:11:20,928 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower552: Last sync leader does not match current leader leader550, need to catch up to -1 01:11:20,928 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower551: Last sync leader does not match current leader leader550, need to catch up to -1 [WARN] [08/19/2024 01:11:20.928] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower551] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@239cbe6e 01:11:20,929 AM [test-akka.actor.default-dispatcher-17] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower551] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [08/19/2024 01:11:20.929] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower552] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@459785ea 01:11:20,929 AM [test-akka.actor.default-dispatcher-17] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower552] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][08/19/2024 01:11:20.929] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower551] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][08/19/2024 01:11:20.929] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower552] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [WARN] [08/19/2024 01:11:20.929] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower553] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4879f10d [INFO] [akkaDeadLetter][08/19/2024 01:11:20.929] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower553] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,929 AM [test-akka.actor.default-dispatcher-15] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower553] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,929 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower553 updated: matchIndex: -1, nextIndex: 0 01:11:20,929 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:20,929 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower552 updated: matchIndex: -1, nextIndex: 0 01:11:20,930 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:20,930 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower551 updated: matchIndex: -1, nextIndex: 0 01:11:20,930 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:20,956 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:11:20,956 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader550 starting 01:11:20,958 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: onShutDown 01:11:20,958 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: Initiating leader transfer 01:11:20,959 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader550: pauseLeader successfully completed - doing transfer 01:11:20,959 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 01:11:20,959 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Attempting to transfer leadership 01:11:20,959 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 01:11:20,959 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower553: onLeaderTransitioning: LeaderTransitioning [leaderId=leader550] 01:11:20,959 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower553: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:20,959 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:20,959 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:20,959 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:11:20,960 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [08/19/2024 01:11:20.960] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower553] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@494c139e [INFO] [akkaDeadLetter][08/19/2024 01:11:20.960] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower553] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower553] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,960 AM [test-akka.actor.default-dispatcher-15] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower553] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,960 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [08/19/2024 01:11:20.961] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower551] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5209bee5 [WARN] [08/19/2024 01:11:20.961] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower552] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7e316886 01:11:20,961 AM [test-akka.actor.default-dispatcher-15] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower552] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][08/19/2024 01:11:20.961] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower552] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:20,961 AM [test-akka.actor.default-dispatcher-15] [INFO] TestActorRef - akkaDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower551] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [akkaDeadLetter][08/19/2024 01:11:20.961] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[akka://test/user/follower551] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 01:11:21,42 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:11:21,42 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:11:21,42 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:11:21,162 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:11:21,162 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:11:21,162 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:11:21,282 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader550: leader transfer aborted 01:11:21,282 AM [test-akka.actor.default-dispatcher-17] [WARN] RaftActorLeadershipTransferCohort - leader550: Failed to transfer leadership in 323.3 ms 01:11:21,282 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550: leader transfer failed - sending PoisonPill 01:11:21,282 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader550 (Leader): Leadership transfer expired 01:11:21,282 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:11:21,282 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower553: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) 01:11:21,282 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader550 and reached -1 (of expected -1) [WARN] [08/19/2024 01:11:21.283] [test-akka.actor.default-dispatcher-16] [akka://test/user/leader550] received dead letter from TestActor[akka://test/user/follower551]: AppendEntriesReply [term=2, success=true, followerId=follower551, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [08/19/2024 01:11:21.283] [test-akka.actor.default-dispatcher-16] [akka://test/user/leader550] received dead letter from TestActor[akka://test/user/follower552]: AppendEntriesReply [term=2, success=true, followerId=follower552, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] [WARN] [08/19/2024 01:11:21.283] [test-akka.actor.default-dispatcher-16] [akka://test/user/leader550] received dead letter from TestActor[akka://test/user/follower553]: AppendEntriesReply [term=2, success=true, followerId=follower553, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:11:21,285 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader550 ending 01:11:21,286 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 01:11:21,286 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower551-notifier554#1594132206] 01:11:21,297 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower551-collector555#1920137906] 01:11:21,298 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower551] 01:11:21,298 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower552-notifier556#-1365497789] 01:11:21,298 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower552-collector557#-177475303] 01:11:21,298 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower552] 01:11:21,299 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower553-notifier558#-1136243998] 01:11:21,299 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower553-collector559#600107663] 01:11:21,299 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower553] 01:11:21,299 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader550-notifier560#-1327545845] 01:11:21,300 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader550-collector561#-1083205295] 01:11:21,300 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader550] 01:11:21,302 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 01:11:21,302 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/19/2024 01:11:21.304] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower563]: org.opendaylight.controller.cluster.common.actor.Monitor@42af2aa6 01:11:21,304 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower563 with journal batch size 1000 01:11:21,305 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: SnapshotOffer called. 01:11:21,305 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Moving last applied index from -1 to -1 01:11:21,305 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=1, votedFor=null 01:11:21,305 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=akka://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=akka://test/user/follower564, votingState=VOTING], PeerInfo [id=follower565, address=akka://test/user/follower565, votingState=NON_VOTING]] 01:11:21,305 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower563 in 115.0 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [08/19/2024 01:11:21.307] [test-akka.actor.default-dispatcher-16] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower564]: org.opendaylight.controller.cluster.common.actor.Monitor@21683144 01:11:21,308 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Recovery completed in 2.558 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:21,310 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower564 with journal batch size 1000 01:11:21,310 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: SnapshotOffer called. 01:11:21,310 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Moving last applied index from -1 to -1 01:11:21,310 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=1, votedFor=null 01:11:21,310 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Updated server config: isVoting: true, peers: [PeerInfo [id=leader562, address=akka://test/user/leader562, votingState=VOTING], PeerInfo [id=follower563, address=akka://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=akka://test/user/follower565, votingState=NON_VOTING]] 01:11:21,310 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower564 in 97.76 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:21,311 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Recovery completed in 1.126 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:21,312 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower565 with journal batch size 1000 01:11:21,312 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: SnapshotOffer called. [WARN] [08/19/2024 01:11:21.312] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower565]: org.opendaylight.controller.cluster.common.actor.Monitor@34d550d6 01:11:21,312 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Moving last applied index from -1 to -1 01:11:21,312 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=1, votedFor=null 01:11:21,312 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Updated server config: isVoting: false, peers: [PeerInfo [id=leader562, address=akka://test/user/leader562, votingState=VOTING], PeerInfo [id=follower564, address=akka://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=akka://test/user/follower563, votingState=VOTING]] 01:11:21,312 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower565 in 65.24 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [08/19/2024 01:11:21.313] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader562]: org.opendaylight.controller.cluster.common.actor.Monitor@11eb5a36 01:11:21,313 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader562 with journal batch size 1000 01:11:21,314 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Recovery completed in 1.375 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:21,314 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: SnapshotOffer called. 01:11:21,314 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Moving last applied index from -1 to -1 01:11:21,314 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=1, votedFor=null 01:11:21,314 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Updated server config: isVoting: true, peers: [PeerInfo [id=follower564, address=akka://test/user/follower564, votingState=VOTING], PeerInfo [id=follower563, address=akka://test/user/follower563, votingState=VOTING], PeerInfo [id=follower565, address=akka://test/user/follower565, votingState=NON_VOTING]] 01:11:21,314 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader562 in 51.06 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:21,314 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Recovery completed in 696.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:21,722 AM [test-akka.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:11:21,722 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.178:25520, Removed) 01:11:21,722 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower): Received ElectionTimeout - switching to Candidate 01:11:21,722 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Election: Candidate has following voting peers: [follower564, follower563] 01:11:21,722 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=2, votedFor=leader562 01:11:21,722 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Starting new election term 2 01:11:21,723 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] to peer follower564 01:11:21,723 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): Sending RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] to peer follower563 01:11:21,723 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:11:21,723 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:21,723 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:21,723 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 01:11:21,723 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Cluster state: [] 01:11:21,723 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:21,723 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:21,723 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:21,723 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:21,723 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=2, votedFor=null 01:11:21,723 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=2, votedFor=null 01:11:21,723 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:21,723 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): In requestVote: RequestVote [term=2, candidateId=leader562, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:21,723 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=2, votedFor=leader562 01:11:21,723 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=2, votedFor=leader562 01:11:21,723 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:21,723 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:21,724 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:11:21,724 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:11:21,724 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Election: Leader has following peers: [follower564, follower563, follower565] 01:11:21,724 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,724 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,724 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,724 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:11:21,724 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader leader562, need to catch up to -1 01:11:21,725 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 2 in "AppendEntries [leaderId=leader562, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:11:21,725 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=2, votedFor=null 01:11:21,725 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader leader562, need to catch up to -1 [WARN] [08/19/2024 01:11:21.725] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower563] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5d922d29 01:11:21,725 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 01:11:21,725 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower564: Last sync leader does not match current leader leader562, need to catch up to -1 01:11:21,725 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,725 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 01:11:21,725 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,725 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower564 updated: matchIndex: -1, nextIndex: 0 01:11:21,725 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:21.725] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower564] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@de4ddce [WARN] [08/19/2024 01:11:21.726] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower565] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4d5ab9f3 01:11:21,769 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:11:21,769 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:11:21,769 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:11:21,769 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:11:21,770 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:11:21,770 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: onRequestLeadership RequestLeadership [requestedFollowerId=follower564, replyTo=Actor[akka://test/user/$a#558055780]] 01:11:21,772 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Initiating leader transfer 01:11:21,772 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 01:11:21,772 AM [test-akka.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader562: pauseLeader successfully completed - doing transfer 01:11:21,772 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 01:11:21,772 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: onLeaderTransitioning: LeaderTransitioning [leaderId=leader562] 01:11:21,772 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Attempting to transfer leadership 01:11:21,772 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,772 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,772 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:11:21,772 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) [WARN] [08/19/2024 01:11:21.772] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower564] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@68defcc3 [WARN] [08/19/2024 01:11:21.772] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower563] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2ddc6fbc 01:11:21,772 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,773 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) [WARN] [08/19/2024 01:11:21.773] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower565] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7b11eebb 01:11:21,773 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): tryToCompleteLeadershipTransfer: followerId: follower564, matchIndex: -1, lastIndex: -1, isVoting: true 01:11:21,773 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Follower's log matches - sending ElectionTimeout 01:11:21,773 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,773 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,773 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:11:21,773 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:11:21,773 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,773 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower565: Lagging 0 entries behind leader leader562 and reached -1 (of expected -1) 01:11:21,773 AM [test-akka.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:11:21,773 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Received TimeoutNow - switching to Candidate 01:11:21,773 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Election: Candidate has following voting peers: [leader562, follower563] 01:11:21,773 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Set currentTerm=3, votedFor=follower564 01:11:21,773 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Starting new election term 3 01:11:21,774 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] to peer leader562 01:11:21,774 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): Sending RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] to peer follower563 01:11:21,774 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:11:21,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:21,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 01:11:21,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:21,774 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 2 - updating term 01:11:21,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=3, votedFor=null 01:11:21,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:21,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Set currentTerm=3, votedFor=follower564 01:11:21,774 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:11:21,774 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leader transfer complete 01:11:21,774 AM [test-akka.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader transfer complete - waiting for new leader 01:11:21,775 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:21,775 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Cluster state: [] 01:11:21,775 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:21,775 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 01:11:21,775 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=3, votedFor=null 01:11:21,775 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): Leadership transfer in progress - processing RequestVote 01:11:21,775 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): In requestVote: RequestVote [term=3, candidateId=follower564, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:21,775 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Set currentTerm=3, votedFor=follower564 01:11:21,776 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:11:21,776 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader562 (Leader) :- Switching from behavior Leader to Follower, election term: 3 01:11:21,776 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 01:11:21,776 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 01:11:21,776 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Election: Leader has following peers: [leader562, follower563, follower565] 01:11:21,776 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,776 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,776 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,776 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 01:11:21,776 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower565 (Follower): Term 3 in "AppendEntries [leaderId=follower564, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 01:11:21,776 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower565: Set currentTerm=3, votedFor=null 01:11:21,776 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - leader562: Last sync leader does not match current leader follower564, need to catch up to -1 01:11:21,777 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower565: Last sync leader does not match current leader follower564, need to catch up to -1 01:11:21,777 AM [test-akka.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader562: leader changed to follower564 01:11:21,777 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader562 updated: matchIndex: -1, nextIndex: 0 01:11:21,777 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower leader562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,777 AM [test-akka.actor.default-dispatcher-5] [INFO] RaftActorLeadershipTransferCohort - leader562: Successfully transferred leadership to follower564 in 4.975 ms 01:11:21,777 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower565 updated: matchIndex: -1, nextIndex: 0 01:11:21,777 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader562: Leadership transferred successfully to follower564 01:11:21,777 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower565: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:21.777] [test-akka.actor.default-dispatcher-5] [akka://test/user/leader562] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@a925a80 [WARN] [08/19/2024 01:11:21.777] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower565] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6d71d5bb 01:11:21,777 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 01:11:21,778 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:21,778 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader follower564, need to catch up to -1 [WARN] [08/19/2024 01:11:21.778] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower563] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4e2453e7 01:11:21,821 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 01:11:21,821 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower563-notifier566#853784866] 01:11:21,832 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower563-collector567#927965571] 01:11:21,832 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower563] 01:11:21,833 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower564-notifier568#-1417255291] 01:11:21,833 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower564-collector569#346422488] 01:11:21,833 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower564] 01:11:21,833 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower565-notifier570#884967671] 01:11:21,834 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower565-collector571#1751117174] 01:11:21,834 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower565] 01:11:21,834 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader562-notifier572#1737282538] 01:11:21,834 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader562-collector573#887178484] 01:11:21,835 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader562] 01:11:21,835 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$a#558055780] 01:11:21,836 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 01:11:21,836 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/19/2024 01:11:21.838] [test-akka.actor.default-dispatcher-5] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower575]: org.opendaylight.controller.cluster.common.actor.Monitor@7f03797f 01:11:21,839 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower575 with journal batch size 1000 01:11:21,840 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: SnapshotOffer called. 01:11:21,840 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Moving last applied index from -1 to -1 01:11:21,840 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=1, votedFor=null 01:11:21,840 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=akka://test/user/leader574, votingState=VOTING], PeerInfo [id=follower577, address=akka://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=akka://test/user/follower576, votingState=VOTING]] 01:11:21,841 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower576 with journal batch size 1000 [WARN] [08/19/2024 01:11:21.841] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower576]: org.opendaylight.controller.cluster.common.actor.Monitor@55d34ef5 01:11:21,842 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: SnapshotOffer called. 01:11:21,842 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Moving last applied index from -1 to -1 01:11:21,842 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=1, votedFor=null 01:11:21,840 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower575 in 101.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:21,842 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Updated server config: isVoting: true, peers: [PeerInfo [id=leader574, address=akka://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=akka://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=akka://test/user/follower577, votingState=NON_VOTING]] 01:11:21,842 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower576 in 126.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:21,843 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Recovery completed in 2.593 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:21,843 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Recovery completed in 1.295 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/19/2024 01:11:21.844] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower577]: org.opendaylight.controller.cluster.common.actor.Monitor@d8d329b 01:11:21,844 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower577 with journal batch size 1000 01:11:21,845 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: SnapshotOffer called. 01:11:21,845 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Moving last applied index from -1 to -1 01:11:21,845 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Set currentTerm=1, votedFor=null 01:11:21,845 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Updated server config: isVoting: false, peers: [PeerInfo [id=leader574, address=akka://test/user/leader574, votingState=VOTING], PeerInfo [id=follower575, address=akka://test/user/follower575, votingState=VOTING], PeerInfo [id=follower576, address=akka://test/user/follower576, votingState=VOTING]] 01:11:21,845 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower577 in 50.48 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:21,845 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Recovery completed in 445.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/19/2024 01:11:21.847] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader574]: org.opendaylight.controller.cluster.common.actor.Monitor@65b30ee5 01:11:21,847 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader574 with journal batch size 1000 01:11:21,847 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: SnapshotOffer called. 01:11:21,847 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Moving last applied index from -1 to -1 01:11:21,847 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Set currentTerm=1, votedFor=null 01:11:21,847 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Updated server config: isVoting: true, peers: [PeerInfo [id=follower575, address=akka://test/user/follower575, votingState=VOTING], PeerInfo [id=follower577, address=akka://test/user/follower577, votingState=NON_VOTING], PeerInfo [id=follower576, address=akka://test/user/follower576, votingState=VOTING]] 01:11:21,847 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader574 in 43.36 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:21,848 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Recovery completed in 389.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:22,172 AM [test-akka.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:11:22,172 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.178:25520, Removed) 01:11:22,172 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower): Received ElectionTimeout - switching to Candidate 01:11:22,172 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Election: Candidate has following voting peers: [follower575, follower576] 01:11:22,172 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Set currentTerm=2, votedFor=leader574 01:11:22,172 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Starting new election term 2 01:11:22,172 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] to peer follower575 01:11:22,172 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): Sending RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] to peer follower576 01:11:22,173 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:11:22,173 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:22,173 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:22,173 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Cluster state: [] 01:11:22,173 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:22,173 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:22,173 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=2, votedFor=null 01:11:22,173 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Cluster state: [] 01:11:22,173 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:22,173 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:22,173 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): In requestVote: RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:22,173 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=2, votedFor=null 01:11:22,173 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Set currentTerm=2, votedFor=leader574 01:11:22,173 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:22,173 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): In requestVote: RequestVote [term=2, candidateId=leader574, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:22,173 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Set currentTerm=2, votedFor=leader574 01:11:22,173 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:22,174 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:11:22,174 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:11:22,174 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Election: Leader has following peers: [follower575, follower577, follower576] 01:11:22,175 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,175 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,175 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower577 (Follower): Term 2 in "AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:11:22,175 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: Set currentTerm=2, votedFor=null 01:11:22,175 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,175 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower577: Last sync leader does not match current leader leader574, need to catch up to -1 01:11:22,175 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower576: Last sync leader does not match current leader leader574, need to catch up to -1 01:11:22,175 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower575: Last sync leader does not match current leader leader574, need to catch up to -1 [WARN] [08/19/2024 01:11:22.175] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower576] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2d1eb28e [WARN] [08/19/2024 01:11:22.175] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower575] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@305c9e97 01:11:22,175 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:11:22,176 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower577 updated: matchIndex: -1, nextIndex: 0 01:11:22,176 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,176 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower576 updated: matchIndex: -1, nextIndex: 0 01:11:22,176 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:22.176] [test-akka.actor.default-dispatcher-15] [akka://test/user/follower577] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3ef34e0a 01:11:22,176 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower575 updated: matchIndex: -1, nextIndex: 0 01:11:22,176 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,201 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:11:22,201 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:11:22,201 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:11:22,201 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower576 starting 01:11:22,202 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: onShutDown 01:11:22,202 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower576 ending 01:11:22,202 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:11:22,202 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:11:22,202 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: onRequestLeadership RequestLeadership [requestedFollowerId=follower576, replyTo=Actor[akka://test/user/$b#-420949544]] 01:11:22,202 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: Initiating leader transfer 01:11:22,203 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] 01:11:22,203 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower577: onLeaderTransitioning: LeaderTransitioning [leaderId=leader574] [WARN] [08/19/2024 01:11:22.203] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower576] received dead letter from TestActor[akka://test/user/leader574]: LeaderTransitioning [leaderId=leader574] 01:11:22,203 AM [test-akka.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader574: pauseLeader successfully completed - doing transfer 01:11:22,203 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Attempting to transfer leadership 01:11:22,203 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,203 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [08/19/2024 01:11:22.204] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower575] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@352daad0 01:11:22,204 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower577: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,204 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [08/19/2024 01:11:22.204] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower577] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2abe45be 01:11:22,204 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:22.204] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower576] received dead letter from TestActor[akka://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [08/19/2024 01:11:22.292] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower576] received dead letter from TestActor[akka://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:11:22,412 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 01:11:22,412 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [08/19/2024 01:11:22.412] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower576] received dead letter from TestActor[akka://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:11:22,531 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader574: leader transfer aborted 01:11:22,531 AM [test-akka.actor.default-dispatcher-17] [WARN] RaftActorLeadershipTransferCohort - leader574: Failed to transfer leadership in 328.6 ms 01:11:22,531 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574: LeadershipTransfer request from follower576 failed 01:11:22,532 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader574 (Leader): Leadership transfer expired 01:11:22,532 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower575: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) 01:11:22,533 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower577: Lagging 0 entries behind leader leader574 and reached -1 (of expected -1) [WARN] [08/19/2024 01:11:22.533] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower576] received dead letter from TestActor[akka://test/user/leader574]: AppendEntries [leaderId=leader574, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:11:22,555 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 01:11:22,556 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower575-notifier578#-1962582290] 01:11:22,567 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower575-collector579#1472032034] 01:11:22,567 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower575] 01:11:22,569 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower576-notifier580#765650627] 01:11:22,569 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower576-collector581#-1947997093] 01:11:22,570 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower576] 01:11:22,570 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower577-notifier582#1516052294] 01:11:22,570 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower577-collector583#1716153063] 01:11:22,570 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower577] 01:11:22,570 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader574-notifier584#-1972913001] 01:11:22,570 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader574-collector585#879302145] 01:11:22,571 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader574] 01:11:22,571 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$b#-420949544] 01:11:22,572 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 01:11:22,572 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/19/2024 01:11:22.573] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower587]: org.opendaylight.controller.cluster.common.actor.Monitor@7b14a58 01:11:22,574 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower587 with journal batch size 1000 01:11:22,574 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: SnapshotOffer called. 01:11:22,574 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to -1 01:11:22,574 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=1, votedFor=null 01:11:22,575 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=akka://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=akka://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=akka://test/user/follower588, votingState=VOTING]] [WARN] [08/19/2024 01:11:22.575] [test-akka.actor.default-dispatcher-15] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower588]: org.opendaylight.controller.cluster.common.actor.Monitor@1d1efc5 01:11:22,576 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower588 with journal batch size 1000 01:11:22,575 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower587 in 203.6 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:22,576 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: SnapshotOffer called. 01:11:22,576 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to -1 01:11:22,576 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=1, votedFor=null 01:11:22,576 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Updated server config: isVoting: true, peers: [PeerInfo [id=leader586, address=akka://test/user/leader586, votingState=VOTING], PeerInfo [id=follower589, address=akka://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower587, address=akka://test/user/follower587, votingState=VOTING]] 01:11:22,576 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower588 in 253.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [08/19/2024 01:11:22.577] [test-akka.actor.default-dispatcher-15] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower589]: org.opendaylight.controller.cluster.common.actor.Monitor@257672eb 01:11:22,577 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Recovery completed in 2.169 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:22,577 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower589 with journal batch size 1000 01:11:22,577 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: SnapshotOffer called. 01:11:22,577 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to -1 01:11:22,577 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=1, votedFor=null 01:11:22,577 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Updated server config: isVoting: false, peers: [PeerInfo [id=leader586, address=akka://test/user/leader586, votingState=VOTING], PeerInfo [id=follower588, address=akka://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=akka://test/user/follower587, votingState=VOTING]] 01:11:22,577 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower589 in 37.94 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:22,578 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Recovery completed in 1.851 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:22,578 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Recovery completed in 380.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:22,579 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader586 with journal batch size 1000 [WARN] [08/19/2024 01:11:22.579] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader586]: org.opendaylight.controller.cluster.common.actor.Monitor@7a0ae716 01:11:22,580 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: SnapshotOffer called. 01:11:22,580 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to -1 01:11:22,580 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=1, votedFor=null 01:11:22,580 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Updated server config: isVoting: true, peers: [PeerInfo [id=follower589, address=akka://test/user/follower589, votingState=NON_VOTING], PeerInfo [id=follower588, address=akka://test/user/follower588, votingState=VOTING], PeerInfo [id=follower587, address=akka://test/user/follower587, votingState=VOTING]] 01:11:22,580 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader586 in 105.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:22,581 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Recovery completed in 642.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:22,942 AM [test-akka.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:11:22,942 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.178:25520, Removed) 01:11:22,942 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower): Received ElectionTimeout - switching to Candidate 01:11:22,942 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Election: Candidate has following voting peers: [follower588, follower587] 01:11:22,942 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=2, votedFor=leader586 01:11:22,942 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Starting new election term 2 01:11:22,943 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] to peer follower588 01:11:22,943 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): Sending RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] to peer follower587 01:11:22,943 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:11:22,943 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:22,943 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Cluster state: [] 01:11:22,943 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:22,943 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:22,943 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=2, votedFor=null 01:11:22,943 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): In requestVote: RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:22,943 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=2, votedFor=leader586 01:11:22,944 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:22,944 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:22,944 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 01:11:22,944 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:22,944 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:22,944 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=2, votedFor=null 01:11:22,944 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote [term=2, candidateId=leader586, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:22,944 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=2, votedFor=leader586 01:11:22,944 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:22,944 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:11:22,944 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:11:22,944 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Election: Leader has following peers: [follower589, follower588, follower587] 01:11:22,945 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,945 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,945 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 2 in "AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:11:22,945 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=2, votedFor=null 01:11:22,945 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader leader586, need to catch up to -1 01:11:22,945 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader leader586, need to catch up to -1 [WARN] [08/19/2024 01:11:22.945] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower588] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6fd04d56 01:11:22,946 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:22.946] [test-akka.actor.default-dispatcher-6] [akka://test/user/follower589] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@70bbd62d 01:11:22,946 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower587: Last sync leader does not match current leader leader586, need to catch up to -1 [WARN] [08/19/2024 01:11:22.946] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower587] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1d35148c 01:11:22,946 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:11:22,947 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: -1, nextIndex: 0 01:11:22,947 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,947 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: -1, nextIndex: 0 01:11:22,947 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,947 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: -1, nextIndex: 0 01:11:22,947 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:22,984 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:11:22,984 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 01:11:22,984 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:22,984 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:22,984 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 01:11:22,984 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:22,984 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower589 01:11:22,984 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower589: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:22,984 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:22,984 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:11:22,984 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Number of entries to be appended = 1 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:11:22,985 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Append entry to log zero 01:11:22,985 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower587 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:22,985 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower587: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Log size is now 1 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower589, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 01:11:22,985 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:22,985 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Number of entries to be appended = 1 01:11:22,985 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:11:22,985 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Append entry to log zero 01:11:22,985 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 01:11:22,985 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Log size is now 1 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:22,985 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower587, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:11:22,985 AM [test-akka.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached -1 (of expected -1) 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: 0, nextIndex: 1 01:11:22,985 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:22,986 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:22,986 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:11:22,986 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Setting last applied to 0 01:11:22,986 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Moving last applied index from -1 to 0 01:11:22,986 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Applying state for log index 0 data zero 01:11:22,986 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [08/19/2024 01:11:22.986] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower589] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1faf269a 01:11:22,986 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: applyState called: zero 01:11:22,986 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Persisting ApplyJournalEntries with index=0 01:11:22,987 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [08/19/2024 01:11:22.987] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower587] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@58183a1 01:11:23,62 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:11:23,62 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:23,182 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:11:23,182 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=leader586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:23,182 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Commit index set to 0 01:11:23,182 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Commit index set to 0 01:11:23,182 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:11:23,182 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:11:23,182 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Setting last applied to 0 01:11:23,182 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to 0 01:11:23,182 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Applying state for log index 0 data zero 01:11:23,182 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: applyState called: zero 01:11:23,182 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Persisting ApplyJournalEntries with index=0 01:11:23,183 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:11:23,183 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:11:23,184 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Setting last applied to 0 01:11:23,184 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Moving last applied index from -1 to 0 01:11:23,184 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Applying state for log index 0 data zero 01:11:23,184 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: applyState called: zero 01:11:23,184 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Persisting ApplyJournalEntries with index=0 01:11:23,186 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 01:11:23,186 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 01:11:23,186 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: onShutDown 01:11:23,186 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Initiating leader transfer 01:11:23,187 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 01:11:23,188 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 01:11:23,188 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader586: pauseLeader successfully completed - doing transfer 01:11:23,188 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Attempting to transfer leadership 01:11:23,188 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,188 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:11:23,188 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,188 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:11:23,188 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,188 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:11:23,189 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower589, matchIndex: 0, lastIndex: 0, isVoting: false 01:11:23,189 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): tryToCompleteLeadershipTransfer: followerId: follower587, matchIndex: 0, lastIndex: 0, isVoting: true 01:11:23,189 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Follower's log matches - sending ElectionTimeout 01:11:23,189 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,189 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:11:23,189 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,189 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): sendAppendEntries: 0 is present for follower follower588 01:11:23,189 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,189 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader leader586 and reached 0 (of expected -1) 01:11:23,189 AM [test-akka.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:11:23,189 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Received TimeoutNow - switching to Candidate 01:11:23,190 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Election: Candidate has following voting peers: [leader586, follower588] 01:11:23,190 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Set currentTerm=3, votedFor=follower587 01:11:23,190 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Starting new election term 3 01:11:23,190 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] to peer leader586 01:11:23,190 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): Sending RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] to peer follower588 01:11:23,190 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:23,190 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:11:23,190 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Cluster state: [] 01:11:23,190 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Candidate in requestVote:RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 01:11:23,190 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leader transfer complete 01:11:23,190 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 3 in "RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2]" message is greater than follower's term 2 - updating term 01:11:23,190 AM [test-akka.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader transfer complete - waiting for new leader 01:11:23,190 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=3, votedFor=null 01:11:23,190 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): In requestVote: RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:23,190 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Set currentTerm=3, votedFor=follower587 01:11:23,190 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:11:23,191 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 01:11:23,191 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:23,191 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 01:11:23,191 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Cluster state: [] 01:11:23,191 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] with higher term appears reachable, updating term. 01:11:23,191 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2]" message is greater than leader's term 2 - switching to Follower 01:11:23,191 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=3, votedFor=null 01:11:23,191 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Election: Leader has following peers: [leader586, follower589, follower588] 01:11:23,191 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): Leadership transfer in progress - processing RequestVote 01:11:23,191 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,191 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): In requestVote: RequestVote [term=3, candidateId=follower587, lastLogIndex=0, lastLogTerm=2] - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 01:11:23,191 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: Set currentTerm=3, votedFor=follower587 01:11:23,191 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,191 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] 01:11:23,191 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,191 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 01:11:23,191 AM [test-akka.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower589 (Follower): Term 3 in "AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 2 - updating term 01:11:23,191 AM [test-akka.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower589: Set currentTerm=3, votedFor=null 01:11:23,191 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader586 (Leader) :- Switching from behavior Leader to Follower, election term: 3 01:11:23,191 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower589: Last sync leader does not match current leader follower587, need to catch up to -1 01:11:23,191 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower589 updated: matchIndex: 0, nextIndex: 1 01:11:23,191 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower589: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,192 AM [test-akka.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - leader586: Last sync leader does not match current leader follower587, need to catch up to -1 01:11:23,192 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader586 updated: matchIndex: 0, nextIndex: 1 01:11:23,192 AM [test-akka.actor.default-dispatcher-14] [DEBUG] RaftActorLeadershipTransferCohort - leader586: leader changed to follower587 01:11:23,192 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower leader586: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,192 AM [test-akka.actor.default-dispatcher-14] [INFO] RaftActorLeadershipTransferCohort - leader586: Successfully transferred leadership to follower587 in 6.260 ms 01:11:23,193 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader586: leader transfer succeeded - sending PoisonPill 01:11:23,193 AM [test-akka.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - leader586: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) [WARN] [08/19/2024 01:11:23.193] [test-akka.actor.default-dispatcher-5] [akka://test/user/leader586] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4e386a67 [WARN] [08/19/2024 01:11:23.193] [test-akka.actor.default-dispatcher-5] [akka://test/user/leader586] received dead letter: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@77efa830 [WARN] [08/19/2024 01:11:23.193] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower589] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4d3226cf 01:11:23,193 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) [WARN] [08/19/2024 01:11:23.193] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower589] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6468ed62 01:11:23,701 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader follower587, need to catch up to -1 01:11:23,702 AM [test-akka.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower589: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 01:11:23,702 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): sendAppendEntries: 0 is present for follower follower588 01:11:23,702 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Sending AppendEntries to follower follower588: AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries: AppendEntries [leaderId=follower587, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Number of entries to be appended = 1 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Append entry to log zero 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [08/19/2024 01:11:23.702] [test-akka.actor.default-dispatcher-17] [akka://test/user/leader586] received dead letter from TestActor[akka://test/user/follower587]: AppendEntries [leaderId=follower587, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [08/19/2024 01:11:23.702] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower588] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3a149c30 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Log size is now 1 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Commit index set to 0 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries returning : AppendEntriesReply [term=3, success=true, followerId=follower588, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader follower587 and reached 0 (of expected -1) 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Setting last applied to 0 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to 0 01:11:23,702 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Applying state for log index 0 data zero 01:11:23,702 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: 0, nextIndex: 1 01:11:23,702 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:23,703 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: applyState called: zero 01:11:23,703 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:23,704 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Persisting ApplyJournalEntries with index=0 [WARN] [08/19/2024 01:11:23.704] [test-akka.actor.default-dispatcher-14] [akka://test/user/follower588] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2c8a4a42 01:11:23,740 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onLeaderTransitioning: LeaderTransitioning [leaderId=leader586] 01:11:23,740 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 01:11:23,740 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower588 starting 01:11:23,741 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onShutDown 01:11:23,741 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/follower588 ending 01:11:23,741 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 01:11:23,741 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower587-notifier590#46333239] 01:11:23,752 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower587-collector591#1210114220] 01:11:23,752 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower587] 01:11:23,753 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower588-notifier592#1359437384] 01:11:23,753 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower588-collector593#1720055468] 01:11:23,754 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower588] 01:11:23,754 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower589-notifier594#-1423422582] 01:11:23,755 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower589-collector595#-400384613] 01:11:23,755 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower589] 01:11:23,755 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader586-notifier596#-181677564] 01:11:23,755 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader586-collector597#-894513666] 01:11:23,755 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader586] 01:11:23,756 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [08/19/2024 01:11:23.758] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader598]: org.opendaylight.controller.cluster.common.actor.Monitor@7a8b80bc 01:11:23,759 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader598 with journal batch size 1000 01:11:23,760 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:23,760 AM [test-akka.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:11:23,760 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower): Received TimeoutNow - switching to Candidate 01:11:23,760 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Election: Candidate has following voting peers: [] 01:11:23,760 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: Set currentTerm=1, votedFor=leader598 01:11:23,760 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Starting new election term 1 01:11:23,760 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 01:11:23,761 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate): Received ElectionTimeout 01:11:23,761 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Leader): Election: Leader has following peers: [] 01:11:23,761 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader598 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 01:11:23,761 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader598 starting 01:11:23,761 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader598: onShutDown 01:11:23,762 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for akka://test/user/leader598 ending 01:11:23,762 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 01:11:23,762 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader598-collector602#779354222] 01:11:23,773 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader598] 01:11:23,773 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 01:11:23,773 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/19/2024 01:11:23.775] [test-akka.actor.default-dispatcher-6] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower604]: org.opendaylight.controller.cluster.common.actor.Monitor@3b85cc6c 01:11:23,776 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower604 with journal batch size 1000 01:11:23,776 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: SnapshotOffer called. 01:11:23,776 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to -1 01:11:23,776 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=1, votedFor=null 01:11:23,776 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=akka://test/user/follower605, votingState=VOTING], PeerInfo [id=follower606, address=akka://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=akka://test/user/leader603, votingState=VOTING]] 01:11:23,776 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower604 in 89.07 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:23,777 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Recovery completed in 911.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/19/2024 01:11:23.779] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower605]: org.opendaylight.controller.cluster.common.actor.Monitor@613ed5a1 01:11:23,780 AM [test-akka.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower605 with journal batch size 1000 01:11:23,781 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: SnapshotOffer called. 01:11:23,781 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Moving last applied index from -1 to -1 01:11:23,781 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=1, votedFor=null 01:11:23,781 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Updated server config: isVoting: true, peers: [PeerInfo [id=follower604, address=akka://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=akka://test/user/follower606, votingState=NON_VOTING], PeerInfo [id=leader603, address=akka://test/user/leader603, votingState=VOTING]] 01:11:23,781 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower605 in 93.50 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:23,782 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Recovery completed in 668.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/19/2024 01:11:23.782] [test-akka.actor.default-dispatcher-11] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower606]: org.opendaylight.controller.cluster.common.actor.Monitor@32563a79 01:11:23,783 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower606 with journal batch size 1000 01:11:23,783 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: SnapshotOffer called. 01:11:23,783 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to -1 01:11:23,783 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Set currentTerm=1, votedFor=null 01:11:23,783 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Updated server config: isVoting: false, peers: [PeerInfo [id=follower605, address=akka://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=akka://test/user/follower604, votingState=VOTING], PeerInfo [id=leader603, address=akka://test/user/leader603, votingState=VOTING]] 01:11:23,783 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower606 in 54.40 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:23,783 AM [test-akka.actor.default-dispatcher-15] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Recovery completed in 432.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/19/2024 01:11:23.787] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader603]: org.opendaylight.controller.cluster.common.actor.Monitor@39510970 01:11:23,787 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader603 with journal batch size 1000 01:11:23,787 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: SnapshotOffer called. 01:11:23,787 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to -1 01:11:23,787 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Set currentTerm=1, votedFor=null 01:11:23,787 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=akka://test/user/follower605, votingState=VOTING], PeerInfo [id=follower604, address=akka://test/user/follower604, votingState=VOTING], PeerInfo [id=follower606, address=akka://test/user/follower606, votingState=NON_VOTING]] 01:11:23,787 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader603 in 42.46 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:23,788 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Recovery completed in 376.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:24,201 AM [test-akka.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:11:24,201 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.178:25520, Removed) 01:11:24,201 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower): Received ElectionTimeout - switching to Candidate 01:11:24,201 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Election: Candidate has following voting peers: [follower605, follower604] 01:11:24,201 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Set currentTerm=2, votedFor=leader603 01:11:24,201 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Starting new election term 2 01:11:24,201 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] to peer follower605 01:11:24,201 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): Sending RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] to peer follower604 01:11:24,201 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Cluster state: [] 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=2, votedFor=null 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): In requestVote: RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Set currentTerm=2, votedFor=leader603 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Cluster state: [] 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=2, votedFor=null 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): In requestVote: RequestVote [term=2, candidateId=leader603, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Set currentTerm=2, votedFor=leader603 01:11:24,202 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:24,202 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:11:24,202 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:11:24,202 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Election: Leader has following peers: [follower605, follower604, follower606] 01:11:24,202 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:24,202 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:24,203 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:24,203 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:11:24,203 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Term 2 in "AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:11:24,203 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Set currentTerm=2, votedFor=null 01:11:24,203 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower606: Last sync leader does not match current leader leader603, need to catch up to -1 01:11:24,203 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: -1, nextIndex: 0 01:11:24,203 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:24,203 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower605: Last sync leader does not match current leader leader603, need to catch up to -1 01:11:24,203 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower605 updated: matchIndex: -1, nextIndex: 0 01:11:24,203 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:24.203] [test-akka.actor.default-dispatcher-16] [akka://test/user/follower605] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@293344c9 01:11:24,203 AM [test-akka.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower604: Last sync leader does not match current leader leader603, need to catch up to -1 01:11:24,203 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: -1, nextIndex: 0 01:11:24,203 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:24.203] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower604] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5d4c6cd1 [WARN] [08/19/2024 01:11:24.204] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower606] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6a70f64a 01:11:24,242 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:11:24,242 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:11:24,243 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:11:24,243 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 01:11:24,244 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:24,244 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower604 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower604: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower606 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower606: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:24,245 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:24,245 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:24,245 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Number of entries to be appended = 1 01:11:24,245 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Number of entries to be appended = 1 01:11:24,245 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:11:24,245 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:11:24,245 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Append entry to log zero 01:11:24,245 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Append entry to log zero 01:11:24,245 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:24,245 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:24,245 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Log size is now 1 01:11:24,245 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Log size is now 1 01:11:24,245 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower606, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:11:24,245 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): handleAppendEntries returning : AppendEntriesReply [term=2, success=true, followerId=follower604, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 01:11:24,245 AM [test-akka.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 01:11:24,245 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached -1 (of expected -1) 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower606 updated: matchIndex: 0, nextIndex: 1 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: 0, nextIndex: 1 01:11:24,245 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:24,246 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:24,246 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [08/19/2024 01:11:24.246] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower606] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2b7b041b [WARN] [08/19/2024 01:11:24.246] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower604] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@35213884 01:11:24,247 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 01:11:24,247 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:11:24,247 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Setting last applied to 0 01:11:24,247 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Moving last applied index from -1 to 0 01:11:24,247 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Applying state for log index 0 data zero 01:11:24,247 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: applyState called: zero 01:11:24,247 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Persisting ApplyJournalEntries with index=0 01:11:24,321 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:11:24,321 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:24,441 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:11:24,441 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Sending AppendEntries to follower follower605: AppendEntries [leaderId=leader603, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]] 01:11:24,441 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Commit index set to 0 01:11:24,441 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Commit index set to 0 01:11:24,442 AM [test-akka.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:11:24,442 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:11:24,442 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:11:24,442 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:11:24,442 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Setting last applied to 0 01:11:24,442 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606 (Follower): Setting last applied to 0 01:11:24,442 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to 0 01:11:24,442 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Moving last applied index from -1 to 0 01:11:24,442 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Applying state for log index 0 data zero 01:11:24,442 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Applying state for log index 0 data zero 01:11:24,442 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: applyState called: zero 01:11:24,442 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: applyState called: zero 01:11:24,442 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: Persisting ApplyJournalEntries with index=0 01:11:24,442 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Persisting ApplyJournalEntries with index=0 01:11:24,446 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 01:11:24,446 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:11:24,446 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:11:24,446 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: onRequestLeadership RequestLeadership [requestedFollowerId=follower605, replyTo=Actor[akka://test/user/$c#-1552398183]] 01:11:24,446 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: Initiating leader transfer 01:11:24,446 AM [test-akka.actor.default-dispatcher-15] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 01:11:24,446 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 01:11:24,446 AM [test-akka.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader603: pauseLeader successfully completed - doing transfer 01:11:24,446 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Attempting to transfer leadership 01:11:24,446 AM [test-akka.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower606: onLeaderTransitioning: LeaderTransitioning [leaderId=leader603] 01:11:24,446 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:24,446 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): sendAppendEntries: 0 is present for follower follower605 01:11:24,446 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:24,447 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Checking sendAppendEntries for follower follower606: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:11:24,447 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:11:24,447 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:11:24,562 AM [test-akka.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:11:24,562 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:11:24,682 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:11:24,682 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:11:24,801 AM [test-akka.actor.default-dispatcher-14] [DEBUG] RaftActorLeadershipTransferCohort - leader603: leader transfer aborted 01:11:24,801 AM [test-akka.actor.default-dispatcher-14] [WARN] RaftActorLeadershipTransferCohort - leader603: Failed to transfer leadership in 355.7 ms 01:11:24,802 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603: LeadershipTransfer request from follower605 failed 01:11:24,802 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader603 (Leader): Leadership transfer expired 01:11:24,802 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower606: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:11:24,802 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower604: Lagging 0 entries behind leader leader603 and reached 0 (of expected -1) 01:11:24,848 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 01:11:24,848 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower604-notifier607#210681198] 01:11:24,859 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower604-collector608#-1813959241] 01:11:24,859 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower604] 01:11:24,859 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower605-notifier609#1222862894] 01:11:24,859 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower605-collector610#-795489639] 01:11:24,859 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower605] 01:11:24,859 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower606-notifier611#-495685343] 01:11:24,859 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower606-collector612#1153472993] 01:11:24,860 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower606] 01:11:24,860 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader603-notifier613#1152913359] 01:11:24,860 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader603-collector614#2110454123] 01:11:24,860 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader603] 01:11:24,860 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$c#-1552398183] 01:11:24,861 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 01:11:24,861 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [08/19/2024 01:11:24.869] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower616]: org.opendaylight.controller.cluster.common.actor.Monitor@6e559af3 01:11:24,869 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower616 with journal batch size 1000 01:11:24,870 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: SnapshotOffer called. 01:11:24,870 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Moving last applied index from -1 to -1 01:11:24,870 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=1, votedFor=null 01:11:24,870 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Updated server config: isVoting: true, peers: [PeerInfo [id=follower618, address=akka://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=akka://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=akka://test/user/leader615, votingState=VOTING]] 01:11:24,870 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower616 in 67.20 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:24,870 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Recovery completed in 716.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/19/2024 01:11:24.971] [test-akka.actor.default-dispatcher-14] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower617]: org.opendaylight.controller.cluster.common.actor.Monitor@49fde460 01:11:24,972 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower617 with journal batch size 1000 01:11:24,972 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: SnapshotOffer called. 01:11:24,972 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Moving last applied index from -1 to -1 01:11:24,972 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=1, votedFor=null 01:11:24,973 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=akka://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=akka://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=leader615, address=akka://test/user/leader615, votingState=VOTING]] 01:11:24,973 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower617 in 81.39 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:24,973 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Recovery completed in 582.1 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/19/2024 01:11:24.973] [test-akka.actor.default-dispatcher-17] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/follower618]: org.opendaylight.controller.cluster.common.actor.Monitor@15837676 01:11:24,974 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower618 with journal batch size 1000 01:11:24,974 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: SnapshotOffer called. 01:11:24,974 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Moving last applied index from -1 to -1 01:11:24,974 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Set currentTerm=1, votedFor=null 01:11:24,974 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Updated server config: isVoting: false, peers: [PeerInfo [id=follower616, address=akka://test/user/follower616, votingState=VOTING], PeerInfo [id=follower617, address=akka://test/user/follower617, votingState=VOTING], PeerInfo [id=leader615, address=akka://test/user/leader615, votingState=VOTING]] 01:11:24,974 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower618 in 61.72 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:24,975 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Recovery completed in 521.7 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/19/2024 01:11:24.976] [test-akka.actor.default-dispatcher-14] [akka://test/user/termination-monitor] received dead letter from TestActor[akka://test/user/leader615]: org.opendaylight.controller.cluster.common.actor.Monitor@73be3641 01:11:24,976 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader615 with journal batch size 1000 01:11:24,977 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: SnapshotOffer called. 01:11:24,977 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Moving last applied index from -1 to -1 01:11:24,977 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=1, votedFor=null 01:11:24,977 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=akka://test/user/follower616, votingState=VOTING], PeerInfo [id=follower618, address=akka://test/user/follower618, votingState=NON_VOTING], PeerInfo [id=follower617, address=akka://test/user/follower617, votingState=VOTING]] 01:11:24,977 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader615 in 59.15 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 01:11:24,977 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Recovery completed in 559.4 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 01:11:25,301 AM [test-akka.actor.default-dispatcher-14] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:11:25,301 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(akka://test@10.30.171.178:25520, Removed) 01:11:25,302 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower): Received ElectionTimeout - switching to Candidate 01:11:25,302 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Election: Candidate has following voting peers: [follower616, follower617] 01:11:25,302 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=2, votedFor=leader615 01:11:25,302 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Starting new election term 2 01:11:25,302 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] to peer follower616 01:11:25,302 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): Sending RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] to peer follower617 01:11:25,302 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:25,302 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:25,302 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Cluster state: [] 01:11:25,302 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Cluster state: [] 01:11:25,302 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:25,302 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Candidate in requestVote:RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:25,302 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:25,302 AM [test-akka.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Term 2 in "RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1]" message is greater than follower's term 1 - updating term 01:11:25,302 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=2, votedFor=null 01:11:25,302 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=2, votedFor=null 01:11:25,302 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): In requestVote: RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:25,302 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): In requestVote: RequestVote [term=2, candidateId=leader615, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:25,302 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Set currentTerm=2, votedFor=leader615 01:11:25,302 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=2, votedFor=leader615 01:11:25,302 AM [test-akka.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:25,302 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): requestVote returning: RequestVoteReply [term=2, voteGranted=true] 01:11:25,302 AM [test-akka.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:11:25,304 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): RaftRPC message received RequestVoteReply [term=2, voteGranted=true], my term is 2 01:11:25,304 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate): handleRequestVoteReply: RequestVoteReply [term=2, voteGranted=true], current voteCount: 1 01:11:25,304 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Election: Leader has following peers: [follower616, follower618, follower617] 01:11:25,304 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:25,304 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:25,304 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:25,304 AM [test-akka.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower616: Last sync leader does not match current leader leader615, need to catch up to -1 01:11:25,304 AM [test-akka.actor.default-dispatcher-16] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:11:25,304 AM [test-akka.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower617: Last sync leader does not match current leader leader615, need to catch up to -1 01:11:25,304 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower616 updated: matchIndex: -1, nextIndex: 0 [WARN] [08/19/2024 01:11:25.304] [test-akka.actor.default-dispatcher-18] [akka://test/user/follower616] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@60afa15b 01:11:25,304 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:25,305 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower617 updated: matchIndex: -1, nextIndex: 0 01:11:25,305 AM [test-akka.actor.default-dispatcher-16] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:25.305] [test-akka.actor.default-dispatcher-17] [akka://test/user/follower617] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@23460cbf 01:11:25,305 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower618 (Follower): Term 2 in "AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]]" message is greater than follower's term 1 - updating term 01:11:25,305 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower618: Set currentTerm=2, votedFor=null 01:11:25,305 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower618: Last sync leader does not match current leader leader615, need to catch up to -1 01:11:25,305 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower618 updated: matchIndex: -1, nextIndex: 0 01:11:25,305 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:25.305] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower618] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1f0f1f2f 01:11:25,329 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 01:11:25,329 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 01:11:25,330 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 01:11:25,330 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower616] 01:11:25,331 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower618] 01:11:25,332 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 01:11:25,332 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 01:11:25,332 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: onRequestLeadership RequestLeadership [requestedFollowerId=follower617, replyTo=Actor[akka://test/user/$d#820304968]] 01:11:25,332 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Initiating leader transfer [WARN] [08/19/2024 01:11:25.332] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower616] received dead letter from TestActor[akka://test/user/leader615]: LeaderTransitioning [leaderId=leader615] 01:11:25,332 AM [test-akka.actor.default-dispatcher-14] [DEBUG] RaftActorLeadershipTransferCohort - leader615: pauseLeader successfully completed - doing transfer 01:11:25,332 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Attempting to transfer leadership 01:11:25,332 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: onLeaderTransitioning: LeaderTransitioning [leaderId=leader615] [WARN] [08/19/2024 01:11:25.332] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower618] received dead letter from TestActor[akka://test/user/leader615]: LeaderTransitioning [leaderId=leader615] 01:11:25,332 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:25,332 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:25.332] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower616] received dead letter from TestActor[akka://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:11:25,332 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:25.332] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower618] received dead letter from TestActor[akka://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:11:25,332 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): tryToCompleteLeadershipTransfer: followerId: follower617, matchIndex: -1, lastIndex: -1, isVoting: true 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Follower's log matches - sending ElectionTimeout 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower618: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leader transfer complete 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader transfer complete - waiting for new leader [WARN] [08/19/2024 01:11:25.333] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower617] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@29ddc7ac [WARN] [08/19/2024 01:11:25.333] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower616] received dead letter from TestActor[akka://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [08/19/2024 01:11:25.333] [test-akka.actor.default-dispatcher-11] [akka://test/user/follower618] received dead letter from TestActor[akka://test/user/leader615]: AppendEntries [leaderId=leader615, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower617: Lagging 0 entries behind leader leader615 and reached -1 (of expected -1) 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Received TimeoutNow - switching to Candidate 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Election: Candidate has following voting peers: [follower616, leader615] 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Set currentTerm=3, votedFor=follower617 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Starting new election term 3 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] to peer follower616 01:11:25,333 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): Sending RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] to peer leader615 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Cluster state: [] 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Candidate in requestVote:RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] with higher term appears reachable, updating term. 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Term 3 in "RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1]" message is greater than leader's term 2 - switching to Follower 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=3, votedFor=null 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): Leadership transfer in progress - processing RequestVote 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): In requestVote: RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Set currentTerm=3, votedFor=follower617 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader): requestVote returning: RequestVoteReply [term=3, voteGranted=true] [WARN] [08/19/2024 01:11:25.334] [test-akka.actor.default-dispatcher-14] [akka://test/user/follower616] received dead letter from TestActor[akka://test/user/follower617]: RequestVote [term=3, candidateId=follower617, lastLogIndex=-1, lastLogTerm=-1] 01:11:25,334 AM [test-akka.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader615 (Leader) :- Switching from behavior Leader to Follower, election term: 3 01:11:25,334 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): RaftRPC message received RequestVoteReply [term=3, voteGranted=true], my term is 3 01:11:25,334 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate): handleRequestVoteReply: RequestVoteReply [term=3, voteGranted=true], current voteCount: 1 01:11:25,334 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Election: Leader has following peers: [follower616, follower618, leader615] 01:11:25,335 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:25,335 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower follower618: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:25.335] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower616] received dead letter from TestActor[akka://test/user/follower617]: AppendEntries [leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] [WARN] [08/19/2024 01:11:25.335] [test-akka.actor.default-dispatcher-5] [akka://test/user/follower618] received dead letter from TestActor[akka://test/user/follower617]: AppendEntries [leaderId=follower617, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, leaderAddress=null, entries=[]] 01:11:25,335 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:11:25,335 AM [test-akka.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 01:11:25,335 AM [test-akka.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - leader615: Last sync leader does not match current leader follower617, need to catch up to -1 01:11:25,335 AM [test-akka.actor.default-dispatcher-14] [DEBUG] RaftActorLeadershipTransferCohort - leader615: leader changed to follower617 01:11:25,335 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader615 updated: matchIndex: -1, nextIndex: 0 01:11:25,335 AM [test-akka.actor.default-dispatcher-14] [INFO] RaftActorLeadershipTransferCohort - leader615: Successfully transferred leadership to follower617 in 3.185 ms 01:11:25,335 AM [test-akka.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader615: Leadership transferred successfully to follower617 01:11:25,335 AM [test-akka.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Leader): Checking sendAppendEntries for follower leader615: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/19/2024 01:11:25.335] [test-akka.actor.default-dispatcher-17] [akka://test/user/leader615] unhandled message from Actor[akka://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@b7dd3ab 01:11:25,382 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 01:11:25,382 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower616-notifier619#2135727048] 01:11:25,383 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower616-collector620#-1938809770] 01:11:25,383 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower617-notifier621#754144167] 01:11:25,384 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower617-collector622#-1852970672] 01:11:25,384 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/follower617] 01:11:25,384 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower618-notifier623#-1598811571] 01:11:25,384 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/follower618-collector624#-750786780] 01:11:25,384 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader615-notifier625#-1253618713] 01:11:25,384 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/leader615-collector626#-1507828195] 01:11:25,384 AM [main] [INFO] TestActorFactory - Killing actor TestActor[akka://test/user/leader615] 01:11:25,384 AM [main] [INFO] TestActorFactory - Killing actor Actor[akka://test/user/$d#820304968] [INFO] [08/19/2024 01:11:25.385] [main] [CoordinatedShutdown(akka://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 01:11:25,385 AM [test-akka.actor.default-dispatcher-17] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [08/19/2024 01:11:25.386] [test-akka.actor.default-dispatcher-18] [akka://test/system/cluster/core/daemon] unhandled message from Actor[akka://test/deadLetters]: Leave(akka://test@10.30.171.178:25520) 01:11:25,387 AM [test-akka.actor.default-dispatcher-18] [INFO] Cluster - Cluster Node [akka://test@10.30.171.178:25520] - Exiting completed 01:11:25,387 AM [test-akka.actor.default-dispatcher-18] [INFO] Cluster - Cluster Node [akka://test@10.30.171.178:25520] - Shutting down... 01:11:25,387 AM [test-akka.actor.default-dispatcher-18] [INFO] Cluster - Cluster Node [akka://test@10.30.171.178:25520] - Successfully shut down [INFO] [08/19/2024 01:11:25.387] [test-akka.actor.internal-dispatcher-3] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.178:25520] - Exiting completed [INFO] [08/19/2024 01:11:25.387] [test-akka.actor.internal-dispatcher-3] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.178:25520] - Shutting down... [INFO] [08/19/2024 01:11:25.387] [test-akka.actor.internal-dispatcher-3] [Cluster(akka://test)] Cluster Node [akka://test@10.30.171.178:25520] - Successfully shut down 01:11:25,388 AM [test-akka.actor.default-dispatcher-18] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. [INFO] [08/19/2024 01:11:25.388] [test-akka.remote.default-remote-dispatcher-7] [akka://test@10.30.171.178:25520/system/remoting-terminator] Shutting down remote daemon. 01:11:25,389 AM [test-akka.actor.default-dispatcher-18] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [08/19/2024 01:11:25.389] [test-akka.remote.default-remote-dispatcher-7] [akka://test@10.30.171.178:25520/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 01:11:25,392 AM [test-akka.actor.default-dispatcher-18] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [INFO] [08/19/2024 01:11:25.392] [test-akka.remote.default-remote-dispatcher-8] [akka://test@10.30.171.178:25520/system/remoting-terminator] Remoting shut down. [DEBUG] [08/19/2024 01:11:25.392] [test-akka.actor.internal-dispatcher-3] [EventStream] shutting down: StandardOutLogger