[DEBUG] [07/01/2025 17:40:34.351] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 05:40:34,351 PM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [07/01/2025 17:40:34.352] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [07/01/2025 17:40:34.352] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [07/01/2025 17:40:34.364] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.164:17355] with UID [-9216296290968126891] [INFO] [07/01/2025 17:40:34.364] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.164:17355] - Starting up, Pekko version [1.0.3] ... 05:40:34,365 PM [test-pekko.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.164:17355] with UID [-9216296290968126891] 05:40:34,365 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.164:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [07/01/2025 17:40:34.367] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.164:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [07/01/2025 17:40:34.367] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.164:17355] - Started up successfully 05:40:34,367 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.164:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 05:40:34,367 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.164:17355] - Started up successfully [INFO] [07/01/2025 17:40:34.368] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.164:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [07/01/2025 17:40:34.368] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.164:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 05:40:34,368 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.164:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 05:40:34,368 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.164:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 05:40:34,377 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testReplicationsWithLaggingFollowerCaughtUpViaAppendEntries starting: sending 2 new payloads 05:40:34,383 PM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower761] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [07/01/2025 17:40:34.383] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower761]: org.opendaylight.controller.cluster.common.actor.Monitor@64ec9e1d [INFO] [pekkoDeadLetter][07/01/2025 17:40:34.383] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower761] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:34,385 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - follower761: Starting recovery with journal batch size 1000 05:40:34,385 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower761: started executor 05:40:34,385 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower761: started executor 05:40:34,386 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower761: metadata journal open: replayFrom=1 applyTo=0 05:40:34,386 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower761: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:34,386 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - follower761: journal open: applyTo=0 05:40:34,386 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: journal writer started 05:40:34,386 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: waiting to receive commands 05:40:34,388 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16767009028061607260/follower761/TermInfo.properties does not exist 05:40:34,388 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower761: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16767009028061607260/follower761/journal-v1-1.log 05:40:34,388 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower761: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16767009028061607260/follower761/metadata-v1-1.log 05:40:34,388 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower761: no eligible files found 05:40:34,389 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower761: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:34,392 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower761: Set currentTerm=0, votedFor=null 05:40:34,392 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower761: Local TermInfo store seeded with TermInfo{term=0} 05:40:34,392 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower761: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@42a52b24 05:40:34,392 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower761: Moving last applied index from -1 to -1 05:40:34,392 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower761: empty journal: appending 0 entries 05:40:34,392 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower761: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:34,392 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower761: journal recovery completedin 191.5 μs with journalIndex=1 [WARN] [07/01/2025 17:40:34.483] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower762]: org.opendaylight.controller.cluster.common.actor.Monitor@47ca371b [INFO] [pekkoDeadLetter][07/01/2025 17:40:34.483] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower762] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:34,483 PM [test-pekko.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower762] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:34,484 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower762: Starting recovery with journal batch size 1000 05:40:34,484 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower762: started executor 05:40:34,484 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower762: started executor 05:40:34,485 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower762: metadata journal open: replayFrom=1 applyTo=0 05:40:34,485 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower762: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:34,485 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower762: journal open: applyTo=0 05:40:34,485 PM [follower762-writer-159] [DEBUG] JournalWriteTask - follower762: journal writer started 05:40:34,485 PM [follower762-writer-159] [DEBUG] JournalWriteTask - follower762: waiting to receive commands [WARN] [07/01/2025 17:40:34.487] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader760]: org.opendaylight.controller.cluster.common.actor.Monitor@5d185276 [INFO] [pekkoDeadLetter][07/01/2025 17:40:34.487] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader760] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:34,487 PM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader760] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:34,487 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - leader760: Starting recovery with journal batch size 1000 05:40:34,487 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader760: started executor 05:40:34,487 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader760: started executor 05:40:34,487 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader760: metadata journal open: replayFrom=1 applyTo=0 05:40:34,487 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader760: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:34,487 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - leader760: journal open: applyTo=0 05:40:34,488 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: journal writer started 05:40:34,488 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: waiting to receive commands 05:40:34,488 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16767009028061607260/follower762/TermInfo.properties does not exist 05:40:34,488 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower762: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16767009028061607260/follower762/journal-v1-1.log 05:40:34,488 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower762: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16767009028061607260/follower762/metadata-v1-1.log 05:40:34,488 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower762: no eligible files found 05:40:34,488 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower762: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:34,489 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16767009028061607260/leader760/TermInfo.properties does not exist 05:40:34,489 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16767009028061607260/leader760/journal-v1-1.log 05:40:34,489 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16767009028061607260/leader760/metadata-v1-1.log 05:40:34,489 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader760: no eligible files found 05:40:34,489 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=5, votedFor=leader760 05:40:34,489 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader760: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:34,491 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower762: Set currentTerm=0, votedFor=null 05:40:34,491 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower762: Local TermInfo store seeded with TermInfo{term=0} 05:40:34,491 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower762: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@c9bb803 05:40:34,491 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower762: Moving last applied index from -1 to -1 05:40:34,491 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower762: empty journal: appending 0 entries 05:40:34,492 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower762: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:34,492 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower762: journal recovery completedin 457.6 μs with journalIndex=1 05:40:34,492 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=5, votedFor=leader760 05:40:34,492 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader760: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader760} 05:40:34,492 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader760: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@3aeed8ce 05:40:34,492 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader760: Moving last applied index from -1 to -1 05:40:34,492 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader760: empty journal: appending 0 entries 05:40:34,492 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader760: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:34,493 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader760: journal recovery completedin 332.6 μs with journalIndex=1 05:40:34,962 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 05:40:34,962 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader760 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.164:17355, Removed) 05:40:34,962 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader760 (Follower): Received ElectionTimeout - switching to Candidate 05:40:34,962 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader760 (Candidate): Election: Candidate has following voting peers: [follower762, follower761] 05:40:34,966 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=6, votedFor=leader760 05:40:34,966 PM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - leader760 (Candidate): Starting new election term 6 05:40:34,966 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader760 (Candidate): Sending RequestVote{term=6, candidateId=leader760, lastLogIndex=-1, lastLogTerm=-1} to peer follower762 05:40:34,966 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader760 (Candidate): Sending RequestVote{term=6, candidateId=leader760, lastLogIndex=-1, lastLogTerm=-1} to peer follower761 05:40:34,967 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader760 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 05:40:34,967 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower762 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 05:40:34,967 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower762 (Follower): Cluster state: [] 05:40:34,967 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower762 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader760, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 05:40:34,967 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower762 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader760, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 05:40:34,967 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower761 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 05:40:34,967 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower761 (Follower): Cluster state: [] 05:40:34,967 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower761 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader760, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 05:40:34,967 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower761 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader760, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 05:40:34,970 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower762: Set currentTerm=6, votedFor=null 05:40:34,970 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower762 (Follower): In requestVote: RequestVote{term=6, candidateId=leader760, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 05:40:34,972 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower761: Set currentTerm=6, votedFor=null 05:40:34,972 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower761 (Follower): In requestVote: RequestVote{term=6, candidateId=leader760, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 05:40:34,975 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower762: Set currentTerm=6, votedFor=leader760 05:40:34,975 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower762 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 05:40:34,975 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader760 (Candidate): RaftRPC message received RequestVoteReply{term=6, voteGranted=true}, my term is 6 05:40:34,975 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader760 (Candidate): handleRequestVoteReply: RequestVoteReply{term=6, voteGranted=true}, current voteCount: 1 05:40:34,975 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader760 (Leader): Election: Leader has following peers: [follower762, follower761] 05:40:34,975 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower762: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:34,976 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower761: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:34,976 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader760 (Candidate) :- Switching from behavior Candidate to Leader, election term: 6 05:40:34,976 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower762: Last sync leader does not match current leader leader760, need to catch up to -1 05:40:34,976 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader760 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower762 updated: matchIndex: -1, nextIndex: 0 [WARN] [07/01/2025 17:40:34.976] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower762] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower762, initialSyncDone=false] 05:40:34,976 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower762: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:34,976 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower761: Set currentTerm=6, votedFor=leader760 05:40:34,976 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower761 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 05:40:34,977 PM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower762] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:34,977 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower761: Last sync leader does not match current leader leader760, need to catch up to -1 05:40:34,977 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader760 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower761 updated: matchIndex: -1, nextIndex: 0 05:40:34,977 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower761: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:34,977 PM [test-pekko.actor.default-dispatcher-11] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower761] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][07/01/2025 17:40:34.977] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower762] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower762] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [07/01/2025 17:40:34.977] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower761] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower761, initialSyncDone=false] [INFO] [pekkoDeadLetter][07/01/2025 17:40:34.977] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower761] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower761] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:34,996 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - Leader created and elected 05:40:34,996 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader760: Persist data index=0 term=6 command=zero 05:40:34,997 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader760: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:34,997 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 05:40:34,997 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower762: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:34,997 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: received 1 commands after 509.1 ms 05:40:34,997 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): sendAppendEntries: 0 is present for follower follower762 05:40:34,997 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Sending AppendEntries to follower follower762: AppendEntries{term=6, leaderId=leader760, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:34,997 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower761: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:34,997 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): sendAppendEntries: 0 is present for follower follower761 05:40:34,997 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Sending AppendEntries to follower follower761: AppendEntries{term=6, leaderId=leader760, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:34,997 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader760: awaiting more completions to resolve 1 synchronized callback(s) 05:40:34,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower761 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader760, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:34,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower761 (Follower): Number of entries to be appended = 1 05:40:34,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower761 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 05:40:34,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower761 (Follower): Append entry to log zero 05:40:34,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower761: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:34,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower761 (Follower): Log size is now 1 05:40:34,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower761 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower761, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:34,997 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: received 1 commands after 611.1 ms 05:40:34,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower761: Lagging 0 entries behind leader leader760 and reached -1 (of expected -1) [WARN] [07/01/2025 17:40:34.997] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower761] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower761, initialSyncDone=true] 05:40:34,998 PM [test-pekko.actor.default-dispatcher-12] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower761] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][07/01/2025 17:40:34.997] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower761] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower761] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:35,0 PM [leader760-writer-160] [DEBUG] RaftStorageCompleter - leader760: 1 completion(s) pending 05:40:35,0 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: completed 1 commands in 3.518 ms 05:40:35,0 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: waiting to receive commands 05:40:35,0 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader760: running 1 completion(s) 05:40:35,0 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader760: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:35,0 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader760: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@4c4342bc 05:40:35,0 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower761 updated: matchIndex: 0, nextIndex: 1 05:40:35,0 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader760 (Leader): Setting last applied to 0 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader760: Moving last applied index from -1 to 0 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader760: Applying state for log index 0 data zero 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader760: applyState called: zero 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower761: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:35,1 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: received 1 commands after 650.9 μs 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader760: Persist data index=1 term=6 command=one 05:40:35,1 PM [leader760-writer-160] [DEBUG] EntryJournalV1 - leader760: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader760: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:35,1 PM [leader760-writer-160] [DEBUG] RaftStorageCompleter - leader760: 1 completion(s) pending 05:40:35,1 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower761 (Follower): Commit index set to 0 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 05:40:35,1 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: completed 1 commands in 162.5 μs 05:40:35,1 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower761: Lagging 0 entries behind leader leader760 and reached 0 (of expected -1) 05:40:35,1 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: received 1 commands after 5.160 μs 05:40:35,1 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower761 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower762: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:35,1 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower761 (Follower): Setting last applied to 0 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): sendAppendEntries: 0 is present for follower follower762 05:40:35,1 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower761: Moving last applied index from -1 to 0 05:40:35,1 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower761: Applying state for log index 0 data zero 05:40:35,1 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower761: applyState called: zero 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Sending AppendEntries to follower follower762: AppendEntries{term=6, leaderId=leader760, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=false}, JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower761: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): sendAppendEntries: 1 is present for follower follower761 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Sending AppendEntries to follower follower761: AppendEntries{term=6, leaderId=leader760, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:35,1 PM [follower761-writer-158] [DEBUG] RaftStorageCompleter - follower761: 1 completion(s) pending 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader760: running 1 completion(s) 05:40:35,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader760: awaiting more completions to resolve 1 synchronized callback(s) 05:40:35,1 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: completed 1 commands in 4.168 ms 05:40:35,1 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: waiting to receive commands 05:40:35,2 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: received 1 commands after 255.3 μs 05:40:35,2 PM [follower761-writer-158] [DEBUG] EntryJournalV1 - follower761: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower761: running 1 completion(s) 05:40:35,2 PM [follower761-writer-158] [DEBUG] RaftStorageCompleter - follower761: 1 completion(s) pending 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower761: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:35,2 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: completed 1 commands in 162.3 μs 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower761: running 1 completion(s) 05:40:35,2 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: waiting to receive commands 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower761 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader760, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower761 (Follower): Number of entries to be appended = 1 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower761 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower761 (Follower): Append entry to log one 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower761: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower761 (Follower): Log size is now 2 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower761 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower761, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:35,2 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: received 1 commands after 264.3 μs 05:40:35,2 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower761: Lagging 0 entries behind leader leader760 and reached 0 (of expected -1) 05:40:35,2 PM [leader760-writer-160] [DEBUG] RaftStorageCompleter - leader760: 1 completion(s) pending 05:40:35,2 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: completed 1 commands in 1.271 ms 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader760: running 1 completion(s) 05:40:35,3 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: waiting to receive commands 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader760: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader760: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@d7e4887 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): sendAppendEntries: 1 is present for follower follower761 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower761 updated: matchIndex: 1, nextIndex: 2 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader760 (Leader): Setting last applied to 1 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader760: Moving last applied index from 0 to 1 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader760: Applying state for log index 1 data one 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader760: applyState called: one 05:40:35,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower761: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:35,3 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: received 1 commands after 349.6 μs 05:40:35,3 PM [leader760-writer-160] [DEBUG] EntryJournalV1 - leader760: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:35,3 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower761 (Follower): Commit index set to 1 05:40:35,3 PM [leader760-writer-160] [DEBUG] RaftStorageCompleter - leader760: 1 completion(s) pending 05:40:35,3 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: completed 1 commands in 134.5 μs 05:40:35,3 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: waiting to receive commands 05:40:35,3 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader760: running 1 completion(s) 05:40:35,3 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower761: Lagging 0 entries behind leader leader760 and reached 1 (of expected -1) 05:40:35,3 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower761 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 05:40:35,3 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower761 (Follower): Setting last applied to 1 05:40:35,3 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower761: Moving last applied index from 0 to 1 05:40:35,3 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower761: Applying state for log index 1 data one 05:40:35,3 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower761: applyState called: one 05:40:35,4 PM [follower761-writer-158] [DEBUG] RaftStorageCompleter - follower761: 1 completion(s) pending 05:40:35,4 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: completed 1 commands in 1.384 ms 05:40:35,4 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: received 1 commands after 6.470 μs 05:40:35,4 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower761: running 1 completion(s) 05:40:35,4 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower761: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 05:40:35,4 PM [follower761-writer-158] [DEBUG] EntryJournalV1 - follower761: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:35,4 PM [follower761-writer-158] [DEBUG] RaftStorageCompleter - follower761: 1 completion(s) pending 05:40:35,4 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: completed 1 commands in 122.8 μs 05:40:35,4 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower761: running 1 completion(s) 05:40:35,4 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: waiting to receive commands 05:40:35,92 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader760 (Leader): sendAppendEntries: 0 is present for follower follower762 05:40:35,93 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader760 (Leader): Sending AppendEntries to follower follower762: AppendEntries{term=6, leaderId=leader760, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=false}, JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 05:40:35,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader760 (Leader): sendAppendEntries: 0 is present for follower follower762 05:40:35,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader760 (Leader): Sending AppendEntries to follower follower762: AppendEntries{term=6, leaderId=leader760, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=false}, JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 05:40:35,213 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower761: Lagging 0 entries behind leader leader760 and reached 1 (of expected -1) 05:40:35,248 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testReplicationsWithLaggingFollowerCaughtUpViaAppendEntries: new entries applied - resuming follower follower762 05:40:35,333 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader760 (Leader): sendAppendEntries: 0 is present for follower follower762 05:40:35,333 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader760 (Leader): Sending AppendEntries to follower follower762: AppendEntries{term=6, leaderId=leader760, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=false}, JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower762 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader760, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=false}, JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower762 (Follower): Number of entries to be appended = 2 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower762 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower762 (Follower): Append entry to log zero 05:40:35,333 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower761: Lagging 0 entries behind leader leader760 and reached 1 (of expected -1) 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower762: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower762 (Follower): Append entry to log one 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower762: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=false} 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower762 (Follower): Log size is now 2 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower762 (Follower): Commit index set to 1 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower762 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower762, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower762: Lagging 0 entries behind leader leader760 and reached 1 (of expected -1) 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower762 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 1 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower762 (Follower): Setting last applied to 0 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower762: Moving last applied index from -1 to 0 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower762: Applying state for log index 0 data zero 05:40:35,333 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower762: applyState called: zero 05:40:35,333 PM [follower762-writer-159] [DEBUG] JournalWriteTask - follower762: received 2 commands after 848.0 ms 05:40:35,334 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader760 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower762 updated: matchIndex: 1, nextIndex: 2 05:40:35,334 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader760: fakeSnapshot purging log to 0 for term 6 05:40:35,334 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader760 (Leader): Checking sendAppendEntries for follower follower762: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 05:40:35,334 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower762 (Follower): Setting last applied to 1 05:40:35,334 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower762: Moving last applied index from 0 to 1 05:40:35,334 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower762: Applying state for log index 1 data one 05:40:35,334 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower762: applyState called: one [WARN] [07/01/2025 17:40:35.334] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower762] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower762, initialSyncDone=true] [INFO] [pekkoDeadLetter][07/01/2025 17:40:35.334] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower762] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower762] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:35,334 PM [test-pekko.actor.default-dispatcher-16] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower762] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:35,340 PM [follower762-writer-159] [DEBUG] RaftStorageCompleter - follower762: 2 completion(s) pending 05:40:35,341 PM [follower762-writer-159] [DEBUG] JournalWriteTask - follower762: completed 2 commands in 7.045 ms 05:40:35,341 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower762: running 2 completion(s) 05:40:35,341 PM [follower762-writer-159] [DEBUG] JournalWriteTask - follower762: received 1 commands after 7.651 μs 05:40:35,341 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower762: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:35,341 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower762: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 05:40:35,341 PM [follower762-writer-159] [DEBUG] EntryJournalV1 - follower762: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:35,341 PM [follower762-writer-159] [DEBUG] RaftStorageCompleter - follower762: 1 completion(s) pending 05:40:35,341 PM [follower762-writer-159] [DEBUG] JournalWriteTask - follower762: completed 1 commands in 188.3 μs 05:40:35,341 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower762: running 1 completion(s) 05:40:35,341 PM [follower762-writer-159] [DEBUG] JournalWriteTask - follower762: waiting to receive commands 05:40:35,453 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower762: Lagging 0 entries behind leader leader760 and reached 1 (of expected -1) 05:40:35,453 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower761: Lagging 0 entries behind leader leader760 and reached 1 (of expected -1) 05:40:35,453 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - follower762: fakeSnapshot purging log to 0 for term 6 05:40:35,453 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower761: fakeSnapshot purging log to 0 for term 6 05:40:35,500 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testReplicationsWithLaggingFollowerCaughtUpViaAppendEntries complete 05:40:35,500 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower761-collector763#1163124587] 05:40:35,511 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower761] 05:40:35,512 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower761: terminating thread VirtualThread[#920,follower761-writer-158]/waiting 05:40:35,512 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: received 1 commands after 508.2 ms 05:40:35,512 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: completed 1 commands in 12.51 μs 05:40:35,512 PM [follower761-writer-158] [DEBUG] JournalWriteTask - follower761: journal writer stopped 05:40:35,512 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower761: journal closed 05:40:35,512 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower761: journal closed 05:40:35,512 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower761: stopped executor with 0 remaining tasks 05:40:35,512 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower761: stopped executor with 0 remaining tasks 05:40:35,513 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower762-collector764#-1722296635] 05:40:35,513 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower762] 05:40:35,513 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower762: terminating thread VirtualThread[#921,follower762-writer-159]/waiting 05:40:35,513 PM [follower762-writer-159] [DEBUG] JournalWriteTask - follower762: received 1 commands after 172.2 ms 05:40:35,513 PM [follower762-writer-159] [DEBUG] JournalWriteTask - follower762: completed 1 commands in 6.400 μs 05:40:35,513 PM [follower762-writer-159] [DEBUG] JournalWriteTask - follower762: journal writer stopped 05:40:35,513 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower762: journal closed 05:40:35,513 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower762: journal closed 05:40:35,513 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower762: stopped executor with 0 remaining tasks 05:40:35,513 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower762: stopped executor with 0 remaining tasks 05:40:35,513 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader760-collector765#-1610023866] 05:40:35,514 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader760] 05:40:35,514 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - leader760: terminating thread VirtualThread[#922,leader760-writer-160]/waiting 05:40:35,514 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: received 1 commands after 510.9 ms 05:40:35,514 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: completed 1 commands in 6.190 μs 05:40:35,514 PM [leader760-writer-160] [DEBUG] JournalWriteTask - leader760: journal writer stopped 05:40:35,514 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader760: journal closed 05:40:35,514 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - leader760: journal closed 05:40:35,514 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader760: stopped executor with 0 remaining tasks 05:40:35,514 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader760: stopped executor with 0 remaining tasks 05:40:35,517 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaInstallSnapshot starting [WARN] [07/01/2025 17:40:35.518] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower770]: org.opendaylight.controller.cluster.common.actor.Monitor@473277ad [INFO] [pekkoDeadLetter][07/01/2025 17:40:35.518] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower770] to Actor[pekko://test/user/termination-monitor] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:35,518 PM [test-pekko.actor.default-dispatcher-15] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower770] to Actor[pekko://test/user/termination-monitor] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:35,519 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower770: Starting recovery with journal batch size 1000 05:40:35,519 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower770: started executor 05:40:35,519 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower770: started executor 05:40:35,520 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower770: metadata journal open: replayFrom=1 applyTo=0 05:40:35,520 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower770: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:35,520 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower770: journal open: applyTo=0 05:40:35,521 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: journal writer started 05:40:35,521 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands [WARN] [07/01/2025 17:40:35.522] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower771]: org.opendaylight.controller.cluster.common.actor.Monitor@6a392de0 [INFO] [pekkoDeadLetter][07/01/2025 17:40:35.522] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower771] to Actor[pekko://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:35,522 PM [test-pekko.actor.default-dispatcher-15] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower771] to Actor[pekko://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:35,522 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower770/TermInfo.properties does not exist 05:40:35,522 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower770: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower770/journal-v1-1.log 05:40:35,522 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower770: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower770/metadata-v1-1.log 05:40:35,522 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower770: no eligible files found 05:40:35,522 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower770: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:35,523 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower771: Starting recovery with journal batch size 1000 05:40:35,523 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower771: started executor 05:40:35,523 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower771: started executor 05:40:35,524 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower771: metadata journal open: replayFrom=1 applyTo=0 05:40:35,524 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower771: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:35,524 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower771: journal open: applyTo=0 05:40:35,524 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: journal writer started 05:40:35,524 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 05:40:35,525 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower770: Set currentTerm=0, votedFor=null 05:40:35,525 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower770: Local TermInfo store seeded with TermInfo{term=0} 05:40:35,525 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower770: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@4429e92c 05:40:35,525 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from -1 to -1 05:40:35,525 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower770: empty journal: appending 0 entries 05:40:35,525 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:35,525 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower771/TermInfo.properties does not exist 05:40:35,525 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower770: journal recovery completedin 204.2 μs with journalIndex=1 05:40:35,525 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower771/journal-v1-1.log 05:40:35,525 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower771/metadata-v1-1.log 05:40:35,525 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower771: no eligible files found 05:40:35,525 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower771: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/01/2025 17:40:35.528] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader769]: org.opendaylight.controller.cluster.common.actor.Monitor@7cbf1d42 [INFO] [pekkoDeadLetter][07/01/2025 17:40:35.528] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader769] to Actor[pekko://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:35,528 PM [test-pekko.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader769] to Actor[pekko://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 05:40:35,529 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower771: Set currentTerm=0, votedFor=null 05:40:35,529 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower771: Local TermInfo store seeded with TermInfo{term=0} 05:40:35,529 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower771: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@691fae20 05:40:35,529 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower771: Moving last applied index from -1 to -1 05:40:35,529 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - leader769: Starting recovery with journal batch size 1000 05:40:35,529 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader769: started executor 05:40:35,529 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower771: empty journal: appending 0 entries 05:40:35,529 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader769: started executor 05:40:35,529 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:35,529 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower771: journal recovery completedin 416.7 μs with journalIndex=1 05:40:35,529 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader769: metadata journal open: replayFrom=1 applyTo=0 05:40:35,530 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader769: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:35,530 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - leader769: journal open: applyTo=0 05:40:35,530 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: journal writer started 05:40:35,530 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:35,532 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/TermInfo.properties does not exist 05:40:35,532 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/journal-v1-1.log 05:40:35,532 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/metadata-v1-1.log 05:40:35,532 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader769: no eligible files found 05:40:35,532 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=5, votedFor=leader769 05:40:35,532 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - leader769: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:35,534 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=5, votedFor=leader769 05:40:35,534 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - leader769: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader769} 05:40:35,534 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@c27f5 05:40:35,534 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from -1 to -1 05:40:35,535 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - leader769: empty journal: appending 0 entries 05:40:35,535 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:35,535 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - leader769: journal recovery completedin 287.6 μs with journalIndex=1 05:40:35,963 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 05:40:35,963 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - leader769 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.164:17355, Removed) 05:40:35,963 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - leader769 (Follower): Received ElectionTimeout - switching to Candidate 05:40:35,963 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader769 (Candidate): Election: Candidate has following voting peers: [follower771, follower770] 05:40:35,967 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=6, votedFor=leader769 05:40:35,968 PM [test-pekko.actor.default-dispatcher-16] [INFO] Candidate - leader769 (Candidate): Starting new election term 6 05:40:35,968 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader769 (Candidate): Sending RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} to peer follower771 05:40:35,968 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader769 (Candidate): Sending RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} to peer follower770 05:40:35,968 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower771 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 05:40:35,968 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader769 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 05:40:35,968 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower770 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 05:40:35,968 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower771 (Follower): Cluster state: [] 05:40:35,968 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower771 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 05:40:35,968 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower771 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 05:40:35,968 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower770 (Follower): Cluster state: [] 05:40:35,968 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower770 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 05:40:35,968 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower770 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 05:40:35,971 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower771: Set currentTerm=6, votedFor=null 05:40:35,971 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower771 (Follower): In requestVote: RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 05:40:35,972 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower770: Set currentTerm=6, votedFor=null 05:40:35,973 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower770 (Follower): In requestVote: RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 05:40:35,974 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower771: Set currentTerm=6, votedFor=leader769 05:40:35,974 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower771 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 05:40:35,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader769 (Candidate): RaftRPC message received RequestVoteReply{term=6, voteGranted=true}, my term is 6 05:40:35,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader769 (Candidate): handleRequestVoteReply: RequestVoteReply{term=6, voteGranted=true}, current voteCount: 1 05:40:35,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Election: Leader has following peers: [follower771, follower770] 05:40:35,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:35,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:35,974 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader769 (Candidate) :- Switching from behavior Candidate to Leader, election term: 6 05:40:35,974 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower771: Last sync leader does not match current leader leader769, need to catch up to -1 [WARN] [07/01/2025 17:40:35.974] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower771] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower771, initialSyncDone=false] 05:40:35,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower771 updated: matchIndex: -1, nextIndex: 0 05:40:35,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:35,975 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower770: Set currentTerm=6, votedFor=leader769 05:40:35,975 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower770 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 05:40:35,975 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower770: Last sync leader does not match current leader leader769, need to catch up to -1 05:40:35,975 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: -1, nextIndex: 0 [WARN] [07/01/2025 17:40:35.975] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower770] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower770, initialSyncDone=false] 05:40:35,975 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:35,988 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - Leader created and elected 05:40:35,988 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Persist data index=0 term=6 command=zero 05:40:35,988 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:35,988 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 05:40:35,988 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:35,988 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 0 is present for follower follower771 05:40:35,988 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower771: AppendEntries{term=6, leaderId=leader769, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:35,988 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:35,988 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 0 is present for follower follower770 05:40:35,988 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:35,988 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: awaiting more completions to resolve 1 synchronized callback(s) 05:40:35,988 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 458.1 ms 05:40:35,988 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower771 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:35,989 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower771 (Follower): Number of entries to be appended = 1 05:40:35,989 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower771 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 05:40:35,989 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:35,989 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower771 (Follower): Append entry to log zero 05:40:35,989 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 05:40:35,989 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower771: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:35,989 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 05:40:35,989 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Append entry to log zero 05:40:35,989 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:35,989 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower771 (Follower): Log size is now 1 05:40:35,989 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Log size is now 1 05:40:35,989 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower771 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower771, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:35,989 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:35,989 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached -1 (of expected -1) 05:40:35,989 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached -1 (of expected -1) 05:40:35,989 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: received 1 commands after 464.6 ms [WARN] [07/01/2025 17:40:35.989] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower771] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower771, initialSyncDone=true] 05:40:35,989 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 468.4 ms [WARN] [07/01/2025 17:40:35.989] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower770] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower770, initialSyncDone=true] 05:40:35,993 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:35,993 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:35,993 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 4.992 ms 05:40:35,994 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@74118a6d 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower771 updated: matchIndex: 0, nextIndex: 1 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 0 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from -1 to 0 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Applying state for log index 0 data zero 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: applyState called: zero 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:35,994 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 251.7 μs 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 0, nextIndex: 1 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Persist data index=1 term=6 command=one 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 05:40:35,994 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:35,994 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 1 is present for follower follower771 05:40:35,994 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower771 (Follower): Commit index set to 0 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): Commit index set to 0 05:40:35,994 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 164.0 μs 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower771: AppendEntries{term=6, leaderId=leader769, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:35,994 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 05:40:35,994 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower771 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:35,994 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 9.980 μs 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 05:40:35,994 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower771 (Follower): Setting last applied to 0 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 0 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 1 is present for follower follower770 05:40:35,994 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower771: Moving last applied index from -1 to 0 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from -1 to 0 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:35,994 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower771: Applying state for log index 0 data zero 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower770: Applying state for log index 0 data zero 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:35,994 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower771: applyState called: zero 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower770: applyState called: zero 05:40:35,994 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: awaiting more completions to resolve 1 synchronized callback(s) 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): Append entry to log one 05:40:35,994 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:35,995 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): Log size is now 2 05:40:35,995 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:35,995 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 05:40:35,995 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower771 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:35,995 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower771 (Follower): Number of entries to be appended = 1 05:40:35,995 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower771 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 05:40:35,995 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:35,995 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower771 (Follower): Append entry to log one 05:40:35,995 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower771: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:35,995 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 5.595 ms 05:40:35,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:35,995 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 2 commands after 2.910 μs 05:40:35,995 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower771 (Follower): Log size is now 2 05:40:35,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:35,995 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower771 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower771, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:35,995 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 05:40:35,995 PM [follower770-writer-161] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:35,995 PM [follower771-writer-162] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 05:40:35,995 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 05:40:35,995 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower771: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:35,995 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 5.985 ms 05:40:35,995 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: received 2 commands after 6.680 μs 05:40:35,995 PM [follower771-writer-162] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:35,996 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@6b4ce413 05:40:35,996 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 1.535 ms 05:40:35,996 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 1 is present for follower follower771 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 1 is present for follower follower770 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 1, nextIndex: 2 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 1 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 0 to 1 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Applying state for log index 1 data one 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: applyState called: one 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader769: fakeSnapshot purging log to 0 for term 6 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 05:40:35,996 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 284.5 μs 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): Commit index set to 1 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 1 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 0 to 1 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower770: Applying state for log index 1 data one 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower770: applyState called: one 05:40:35,996 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:35,996 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:35,996 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower770: fakeSnapshot purging log to 0 for term 6 05:40:35,996 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower771 updated: matchIndex: 1, nextIndex: 2 05:40:35,996 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 147.8 μs 05:40:35,996 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:35,996 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 05:40:35,996 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:35,996 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower771 (Follower): Commit index set to 1 05:40:35,996 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 05:40:35,996 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower771 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 05:40:35,996 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower771 (Follower): Setting last applied to 1 05:40:35,996 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower771: Moving last applied index from 0 to 1 05:40:35,996 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower771: Applying state for log index 1 data one 05:40:35,996 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower771: applyState called: one 05:40:35,996 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - follower771: fakeSnapshot purging log to 0 for term 6 05:40:35,996 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 2 completion(s) pending 05:40:35,996 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 2 commands in 1.738 ms 05:40:35,996 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower770: running 2 completion(s) 05:40:35,996 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 5.211 μs 05:40:35,997 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 05:40:35,997 PM [follower770-writer-161] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:35,997 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:35,997 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 66.35 μs 05:40:35,997 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:35,997 PM [follower771-writer-162] [DEBUG] RaftStorageCompleter - follower771: 2 completion(s) pending 05:40:35,997 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:35,997 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower771: running 2 completion(s) 05:40:35,997 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower771: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 05:40:35,997 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: completed 2 commands in 1.608 ms 05:40:35,997 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: received 1 commands after 4.060 μs 05:40:35,997 PM [follower771-writer-162] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:35,997 PM [follower771-writer-162] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 05:40:35,997 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 05:40:35,997 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 138.0 μs 05:40:35,997 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 05:40:36,212 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 05:40:36,212 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 05:40:36,332 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 05:40:36,453 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 05:40:36,572 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 05:40:36,645 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Persist data index=2 term=6 command=two 05:40:36,645 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 05:40:36,646 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=two}, logIndex: 2, isSendImmediate: true 05:40:36,646 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 05:40:36,646 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 649.6 ms 05:40:36,646 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 05:40:36,646 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 2 is present for follower follower770 05:40:36,646 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 05:40:36,646 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: awaiting more completions to resolve 1 synchronized callback(s) 05:40:36,646 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 05:40:36,646 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 05:40:36,646 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 05:40:36,646 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Append entry to log two 05:40:36,646 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 05:40:36,646 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Log size is now 2 05:40:36,646 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=2, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:36,646 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 05:40:36,646 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 649.8 ms 05:40:36,648 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,648 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 1.705 ms 05:40:36,648 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,648 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,648 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=false} 05:40:36,648 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,648 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 2.487 ms 05:40:36,648 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,648 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:36,648 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=true} 05:40:36,648 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@2bbc302b 05:40:36,648 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 2, nextIndex: 3 05:40:36,648 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 05:40:36,648 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 2 05:40:36,648 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 1 to 2 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Applying state for log index 2 data two 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: applyState called: two 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader769: fakeSnapshot purging log to 1 for term 6 05:40:36,649 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 512.1 μs 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 1 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Persist data index=3 term=6 command=three 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 05:40:36,649 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=3] 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=three}, logIndex: 3, isSendImmediate: true 05:40:36,649 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: 2, leaderLastIndex: 3, leaderSnapShotIndex: 1 05:40:36,649 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 219.6 μs 05:40:36,649 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 4.310 μs 05:40:36,649 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): Commit index set to 2 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 3, leaderLastIndex: 3, leaderSnapShotIndex: 1 05:40:36,649 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 3 is present for follower follower770 05:40:36,649 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 2 05:40:36,649 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 2 05:40:36,649 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 1 to 2 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 05:40:36,649 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower770: Applying state for log index 2 data two 05:40:36,649 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower770: applyState called: two 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,649 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: awaiting more completions to resolve 1 synchronized callback(s) 05:40:36,649 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower770: fakeSnapshot purging log to 1 for term 6 05:40:36,649 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 1.295 ms 05:40:36,650 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 05:40:36,650 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 05:40:36,650 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 05:40:36,650 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): Append entry to log three 05:40:36,650 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 05:40:36,650 PM [follower770-writer-161] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=3] 05:40:36,650 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): Log size is now 2 05:40:36,650 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,650 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=3, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:36,650 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 148.6 μs 05:40:36,650 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 05:40:36,650 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 2.680 μs 05:40:36,650 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,651 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,651 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,651 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 2.102 ms 05:40:36,651 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:36,651 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=true} 05:40:36,651 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=2, lastAppliedTerm=6, lastIndex=3, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=1, mandatoryTrim=false] 05:40:36,651 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader769: captured snapshot at lastSequenceNumber: 4 05:40:36,651 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Persising snapshot at EntryInfo[index=2, term=6]/EntryInfo[index=3, term=6] 05:40:36,652 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: support() called 05:40:36,652 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Removed in-memory snapshotted entries, adjusted snaphsotIndex: 1 and term: 6 05:40:36,652 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@7cd6220e 05:40:36,652 PM [leader769-%d328628] [DEBUG] RaftStorage - leader769: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-26E42E23.tmp 05:40:36,652 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 3 is present for follower follower770 05:40:36,652 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 3, nextIndex: 4 05:40:36,652 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2 05:40:36,652 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 3 05:40:36,652 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 2 to 3 05:40:36,652 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Applying state for log index 3 data three 05:40:36,652 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: applyState called: three 05:40:36,652 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,653 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 2.780 ms 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,653 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 4, leaderLastIndex: 3, leaderSnapShotIndex: 1 05:40:36,653 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=false} 05:40:36,653 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 1.386 ms 05:40:36,653 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Persist data index=4 term=6 command=four 05:40:36,653 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 05:40:36,653 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=four}, logIndex: 4, isSendImmediate: true 05:40:36,653 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=4] 05:40:36,653 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: 2, leaderLastIndex: 4, leaderSnapShotIndex: 1 05:40:36,653 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Commit index set to 3 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 3 (of expected -1) 05:40:36,653 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 205.9 μs 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): 2, lastIndex(): 3 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 3 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 2 to 3 05:40:36,653 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 4.820 μs 05:40:36,653 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 4, leaderLastIndex: 4, leaderSnapShotIndex: 1 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower770: Applying state for log index 3 data three 05:40:36,653 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 4 is present for follower follower770 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower770: applyState called: three 05:40:36,653 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 05:40:36,653 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,653 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: awaiting more completions to resolve 1 synchronized callback(s) 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 05:40:36,653 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 565.9 μs 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 3, entries to be added from: 0 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Append entry to log four 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Log size is now 3 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=4, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 3 (of expected -1) 05:40:36,653 PM [follower770-writer-161] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=4] 05:40:36,653 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,653 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 183.4 μs 05:40:36,653 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,653 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 2.090 μs 05:40:36,657 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,657 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 3.492 ms 05:40:36,657 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,657 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,657 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,657 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=false} 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=true} 05:40:36,657 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 4.035 ms 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@2ff562c1 05:40:36,657 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 4 is present for follower follower770 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 4, nextIndex: 5 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 4, lastAppliedIndex: 3 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 4 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 3 to 4 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Applying state for log index 4 data four 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: applyState called: four 05:40:36,657 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 5, leaderLastIndex: 4, leaderSnapShotIndex: 1 05:40:36,657 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): Commit index set to 4 05:40:36,657 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 4 (of expected -1) 05:40:36,657 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 4, context.getLastApplied(): 3, lastIndex(): 4 05:40:36,657 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 4 05:40:36,657 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 3 to 4 05:40:36,657 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower770: Applying state for log index 4 data four 05:40:36,657 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 292.9 μs 05:40:36,657 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower770: applyState called: four 05:40:36,658 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 635.6 μs 05:40:36,658 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=5] 05:40:36,658 PM [follower770-writer-161] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=5] 05:40:36,658 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: Persist data index=5 term=6 command=five 05:40:36,658 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,658 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 05:40:36,658 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,658 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 101.6 μs 05:40:36,658 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,658 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=five}, logIndex: 5, isSendImmediate: true 05:40:36,658 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: 2, leaderLastIndex: 5, leaderSnapShotIndex: 1 05:40:36,658 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 248.6 μs 05:40:36,658 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,658 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 4.560 μs 05:40:36,658 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 5, leaderLastIndex: 5, leaderSnapShotIndex: 1 05:40:36,658 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 5 is present for follower follower770 05:40:36,658 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=4, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 05:40:36,658 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,658 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: awaiting more completions to resolve 1 synchronized callback(s) 05:40:36,658 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=4, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 05:40:36,658 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 05:40:36,658 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 4, entries to be added from: 0 05:40:36,658 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower770 (Follower): Append entry to log five 05:40:36,658 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 05:40:36,658 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower770 (Follower): Log size is now 4 05:40:36,658 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=5, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:36,658 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 4 (of expected -1) 05:40:36,658 PM [leader769-%d328628] [DEBUG] RaftStorage - leader769: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-26E42E23.v1 05:40:36,658 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 340.8 μs 05:40:36,658 PM [leader769-%d328628] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-26E42E23.v1 05:40:36,658 PM [leader769-%d328628] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/TermInfo.properties 05:40:36,658 PM [leader769-%d328628] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/journal-v1-1.log 05:40:36,658 PM [leader769-%d328628] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/metadata-v1-1.log 05:40:36,659 PM [leader769-%d328628] [DEBUG] RaftStorage - leader769: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-07-01T17:40:36.652488227Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-26E42E23.v1, size=510, entryCompress=NONE, stateCompress=NONE} 05:40:36,659 PM [leader769-%d328628] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,659 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,659 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader769: snapshot is durable as of 2025-07-01T17:40:36.652488227Z 05:40:36,659 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader769: Snapshot success - sequence number: 4 05:40:36,659 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: awaiting more completions to resolve 1 synchronized callback(s) 05:40:36,659 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,659 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,659 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 1.301 ms 05:40:36,659 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,659 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,659 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=true} 05:40:36,659 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 1.639 ms 05:40:36,659 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@15a22266 05:40:36,659 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 5.180 μs 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 5, nextIndex: 6 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 5, lastAppliedIndex: 4 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 5 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 4 to 5 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: Applying state for log index 5 data five 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader769: applyState called: five 05:40:36,660 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=4, applyTo=5] 05:40:36,660 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 6, leaderLastIndex: 5, leaderSnapShotIndex: 1 05:40:36,660 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 203.4 μs 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,660 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,660 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=false} 05:40:36,660 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 4.210 μs 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorSnapshotMessageSupport - leader769: SnapshotComplete received 05:40:36,660 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower770 (Follower): Commit index set to 5 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: Persist data index=6 term=6 command=six 05:40:36,660 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 5 (of expected -1) 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=true} 05:40:36,660 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 5, context.getLastApplied(): 4, lastIndex(): 5 05:40:36,660 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 5 05:40:36,660 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 4 to 5 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=six}, logIndex: 6, isSendImmediate: true 05:40:36,660 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower770: Applying state for log index 5 data five 05:40:36,660 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower770: applyState called: five 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: 2, leaderLastIndex: 6, leaderSnapShotIndex: 1 05:40:36,660 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=4, applyTo=6] 05:40:36,660 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,660 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 219.7 μs 05:40:36,660 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 653.7 μs 05:40:36,660 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 4.550 μs 05:40:36,660 PM [follower770-writer-161] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=6] 05:40:36,660 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,660 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 104.2 μs 05:40:36,660 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,660 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 6, leaderLastIndex: 6, leaderSnapShotIndex: 1 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 6 is present for follower follower770 05:40:36,660 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=5, prevLogTerm=6, leaderCommit=5, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 05:40:36,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=5, prevLogTerm=6, leaderCommit=5, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 05:40:36,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 05:40:36,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 5, entries to be added from: 0 05:40:36,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): Append entry to log six 05:40:36,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=true} 05:40:36,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): Log size is now 5 05:40:36,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=6, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:36,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 5 (of expected -1) 05:40:36,660 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 255.2 μs 05:40:36,661 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,661 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: awaiting more completions to resolve 1 synchronized callback(s) 05:40:36,662 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,662 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,662 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 1.337 ms 05:40:36,662 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 1.760 ms 05:40:36,662 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,662 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=true} 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@247d72f5 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 6 is present for follower follower770 05:40:36,662 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,662 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=false} 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 6, nextIndex: 7 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 6, lastAppliedIndex: 5 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 6 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 5 to 6 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: Applying state for log index 6 data six 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader769: applyState called: six 05:40:36,662 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 7, leaderLastIndex: 6, leaderSnapShotIndex: 1 05:40:36,662 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 314.3 μs 05:40:36,663 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=4, applyTo=7] 05:40:36,663 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,663 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 88.43 μs 05:40:36,663 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): Commit index set to 6 05:40:36,663 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:36,663 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 6 (of expected -1) 05:40:36,663 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 6, context.getLastApplied(): 5, lastIndex(): 6 05:40:36,663 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 6 05:40:36,663 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 5 to 6 05:40:36,663 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower770: Applying state for log index 6 data six 05:40:36,663 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower770: applyState called: six 05:40:36,663 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,663 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 1.518 ms 05:40:36,663 PM [follower770-writer-161] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=7] 05:40:36,663 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,664 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,664 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 94.23 μs 05:40:36,664 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,746 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-26E42E23.v1 05:40:36,746 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/TermInfo.properties 05:40:36,746 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/journal-v1-1.log 05:40:36,746 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/metadata-v1-1.log 05:40:36,746 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: picked SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-07-01T17:40:36.652488227Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-26E42E23.v1, size=510, entryCompress=NONE, stateCompress=NONE} as the latest file 05:40:36,747 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaInstallSnapshot: sending 1 more payload to trigger second snapshot 05:40:36,747 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Persist data index=7 term=6 command=seven 05:40:36,748 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=7, term=6, command=seven, pending=true} 05:40:36,748 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=seven}, logIndex: 7, isSendImmediate: true 05:40:36,748 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: 2, leaderLastIndex: 7, leaderSnapShotIndex: 1 05:40:36,748 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 84.46 ms 05:40:36,748 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 7, leaderLastIndex: 7, leaderSnapShotIndex: 1 05:40:36,748 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 7 is present for follower follower770 05:40:36,748 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=6, prevLogTerm=6, leaderCommit=6, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=7, term=6, command=seven, pending=true}]} 05:40:36,748 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: awaiting more completions to resolve 1 synchronized callback(s) 05:40:36,748 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=6, prevLogTerm=6, leaderCommit=6, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=7, term=6, command=seven, pending=true}]} 05:40:36,748 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 05:40:36,748 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 6, entries to be added from: 0 05:40:36,748 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Append entry to log seven 05:40:36,748 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=7, term=6, command=seven, pending=true} 05:40:36,748 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Log size is now 6 05:40:36,748 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=7, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:36,748 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 6 (of expected -1) 05:40:36,748 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 84.81 ms 05:40:36,750 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,750 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,750 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,750 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 1.896 ms 05:40:36,750 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 2.529 ms 05:40:36,750 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,750 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=7, term=6, command=seven, pending=true} 05:40:36,750 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:36,750 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,750 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=7, term=6, command=seven, pending=false} 05:40:36,750 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=6, lastAppliedTerm=6, lastIndex=7, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=1, mandatoryTrim=false] 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader769: captured snapshot at lastSequenceNumber: 8 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Persising snapshot at EntryInfo[index=6, term=6]/EntryInfo[index=7, term=6] 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: support() called 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader769: log size 6 exceeds the snapshot batch count 4 - doing snapshotPreCommit with index 6 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Removed in-memory snapshotted entries, adjusted snaphsotIndex: 6 and term: 6 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@4a885f62 05:40:36,751 PM [leader769-%d328629] [DEBUG] RaftStorage - leader769: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-2CCAFE8C.tmp 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 7, nextIndex: 8 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 7, lastAppliedIndex: 6 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 7 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 6 to 7 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Applying state for log index 7 data seven 05:40:36,751 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: applyState called: seven 05:40:36,752 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 8, leaderLastIndex: 7, leaderSnapShotIndex: 6 05:40:36,752 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 1.223 ms 05:40:36,752 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): Commit index set to 7 05:40:36,752 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 7 (of expected -1) 05:40:36,752 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 7, context.getLastApplied(): 6, lastIndex(): 7 05:40:36,752 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 7 05:40:36,752 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=4, applyTo=8] 05:40:36,752 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 6 to 7 05:40:36,752 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower770: Applying state for log index 7 data seven 05:40:36,752 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,752 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower770: applyState called: seven 05:40:36,752 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 274.0 μs 05:40:36,752 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:36,752 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,752 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 1.697 ms 05:40:36,752 PM [follower770-writer-161] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=8] 05:40:36,752 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,752 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 152.5 μs 05:40:36,752 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,752 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,754 PM [leader769-%d328629] [DEBUG] RaftStorage - leader769: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-2CCAFE8C.v1 05:40:36,754 PM [leader769-%d328629] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-2CCAFE8C.v1 05:40:36,754 PM [leader769-%d328629] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-26E42E23.v1 05:40:36,754 PM [leader769-%d328629] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/TermInfo.properties 05:40:36,754 PM [leader769-%d328629] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/journal-v1-1.log 05:40:36,754 PM [leader769-%d328629] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/metadata-v1-1.log 05:40:36,754 PM [leader769-%d328629] [DEBUG] RaftStorage - leader769: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-07-01T17:40:36.652488227Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-26E42E23.v1, size=510, entryCompress=NONE, stateCompress=NONE} 05:40:36,754 PM [leader769-%d328629] [DEBUG] RaftStorage - leader769: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-07-01T17:40:36.751500940Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-2CCAFE8C.v1, size=578, entryCompress=NONE, stateCompress=NONE} 05:40:36,754 PM [leader769-%d328629] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,754 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,754 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: snapshot is durable as of 2025-07-01T17:40:36.751500940Z 05:40:36,754 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader769: Snapshot success - sequence number: 8 05:40:36,755 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorSnapshotMessageSupport - leader769: SnapshotComplete received 05:40:36,755 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 2.566 ms 05:40:36,755 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=8, applyTo=8] 05:40:36,755 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,755 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 184.1 μs 05:40:36,755 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,755 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:36,933 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 7 (of expected -1) 05:40:36,950 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-2CCAFE8C.v1 05:40:36,950 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/TermInfo.properties 05:40:36,950 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/journal-v1-1.log 05:40:36,950 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/metadata-v1-1.log 05:40:36,950 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: picked SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-07-01T17:40:36.751500940Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-2CCAFE8C.v1, size=578, entryCompress=NONE, stateCompress=NONE} as the latest file 05:40:36,952 PM [main] [DEBUG] PeerInfos - leader769: Updated server config: isVoting: true, peers: [PeerInfo [id=follower771, address=pekko://test/user/follower771, votingState=NON_VOTING], PeerInfo [id=follower770, address=pekko://test/user/follower770, votingState=NON_VOTING]] 05:40:36,952 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader769: Persist data index=8 term=6 command=VotingConfig{serverInfo=[ServerInfo[peerId=leader769, isVoting=true], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=follower771, isVoting=false]]} 05:40:36,952 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader769, isVoting=true], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=follower771, isVoting=false]]}, pending=true} 05:40:36,952 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=serverConfig}, logIndex: 8, isSendImmediate: true 05:40:36,952 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: 2, leaderLastIndex: 8, leaderSnapShotIndex: 6 05:40:36,952 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 197.3 ms 05:40:36,952 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 8, leaderLastIndex: 8, leaderSnapShotIndex: 6 05:40:36,952 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 8 is present for follower follower770 05:40:36,952 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=7, prevLogTerm=6, leaderCommit=7, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader769, isVoting=true], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=follower771, isVoting=false]]}, pending=true}]} 05:40:36,952 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader769: awaiting more completions to resolve 1 synchronized callback(s) 05:40:36,952 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=7, prevLogTerm=6, leaderCommit=7, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader769, isVoting=true], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=follower771, isVoting=false]]}, pending=true}]} 05:40:36,953 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 05:40:36,953 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 7, entries to be added from: 0 05:40:36,953 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=leader769, isVoting=true], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=follower771, isVoting=false]]} 05:40:36,953 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader769, isVoting=true], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=follower771, isVoting=false]]}, pending=true} 05:40:36,953 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower770: Updated server config: isVoting: false, peers: [PeerInfo [id=leader769, address=pekko://test/user/leader769, votingState=VOTING], PeerInfo [id=follower771, address=pekko://test/user/follower771, votingState=NON_VOTING]] 05:40:36,953 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): Log size is now 7 05:40:36,953 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 200.4 ms 05:40:36,953 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=8, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:36,953 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 7 (of expected -1) 05:40:36,960 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,960 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,960 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 8.092 ms 05:40:36,960 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:36,960 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,960 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 7.577 ms 05:40:36,960 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:36,960 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,960 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader769, isVoting=true], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=follower771, isVoting=false]]}, pending=true} 05:40:36,960 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader769, isVoting=true], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=follower771, isVoting=false]]}, pending=false} 05:40:36,960 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 8, lastAppliedIndex: 7 05:40:36,960 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 8 05:40:36,960 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 7 to 8 05:40:36,961 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader769: Applying state for log index 8 data VotingConfig{serverInfo=[ServerInfo[peerId=leader769, isVoting=true], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=follower771, isVoting=false]]} 05:40:36,961 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader769: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@1b4482da 05:40:36,961 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 323.6 μs 05:40:36,961 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 8, nextIndex: 9 05:40:36,961 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 9, leaderLastIndex: 8, leaderSnapShotIndex: 6 05:40:36,961 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=8, applyTo=9] 05:40:36,961 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:36,961 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:36,961 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 224.1 μs 05:40:36,961 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:36,961 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): Commit index set to 8 05:40:36,961 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 8 (of expected -1) 05:40:36,961 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 8, context.getLastApplied(): 7, lastIndex(): 8 05:40:36,961 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 8 05:40:36,961 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 7 to 8 05:40:36,961 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower770: Applying state for log index 8 data VotingConfig{serverInfo=[ServerInfo[peerId=leader769, isVoting=true], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=follower771, isVoting=false]]} 05:40:36,961 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 841.1 μs 05:40:36,961 PM [follower770-writer-161] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=9] 05:40:36,961 PM [follower770-writer-161] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 05:40:36,961 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 155.7 μs 05:40:36,961 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 05:40:36,961 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 05:40:37,3 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - verifyInstallSnapshotToLaggingFollower starting 05:40:37,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower771: Lagging -2 entries behind leader leader769 and reached 1 (of expected -1) 05:40:37,53 PM [test-pekko.actor.default-dispatcher-5] [WARN] AbstractLeader - leader769 (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply{term=6, success=true, followerId=follower771, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, timeSinceLastActivity : 840, lastApplied : 8, commitIndex : 8 05:40:37,53 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractLeader - leader769 (Leader): Initiating install snapshot to follower follower771: follower nextIndex: 2, leader snapshotIndex: 6, leader lastIndex: 8, leader log size: 2 05:40:37,53 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader769: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=8, lastAppliedTerm=6, lastIndex=8, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower771 05:40:37,53 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - leader769: lastSequenceNumber prior to capture: 10 05:40:37,53 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader769: support() called 05:40:37,54 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower771: Lagging -2 entries behind leader leader769 and reached 1 (of expected -1) 05:40:37,54 PM [leader769-%d328630] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:37,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:37,54 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader769: Persising snapshot at EntryInfo[index=8, term=6]/EntryInfo[index=8, term=6] 05:40:37,54 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader769: support() called 05:40:37,54 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader769: Removed in-memory snapshotted entries, adjusted snaphsotIndex: 6 and term: 6 05:40:37,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader769 (Leader): sendInstallSnapshot 05:40:37,54 PM [leader769-%d328631] [DEBUG] RaftStorage - leader769: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D95-03416D49.tmp 05:40:37,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] LeaderInstallSnapshotState - leader769 (Leader): Snapshot 396 bytes, total chunks to send: 1 05:40:37,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] LeaderInstallSnapshotState - leader769 (Leader): Next chunk: total length=396, offset=0, size=396, hashCode=-153134944 05:40:37,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader769 (Leader): next snapshot chunk size for follower follower771: 396 05:40:37,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader769 (Leader): InstallSnapshot sent to follower Vector(user, follower771), Chunk: 1/1 05:40:37,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): handleInstallSnapshot: InstallSnapshot{term=6, leaderId=leader769, lastIncludedIndex=8, lastIncludedTerm=6, datasize=396, chunk=1/1, lastChunkHashCode=OptionalInt[-1], votingConfig=VotingConfig{serverInfo=[ServerInfo[peerId=follower771, isVoting=false], ServerInfo[peerId=follower770, isVoting=false], ServerInfo[peerId=leader769, isVoting=true]]}} 05:40:37,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower771: Lagging 7 entries behind leader leader769 and reached 1 (of expected -1) 05:40:37,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotTracker - follower771 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 05:40:37,55 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower771 (Follower): Snapshot received from leader: leader769 05:40:37,55 PM [test-pekko.actor.default-dispatcher-15] [INFO] SnapshotManager - follower771: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=396}} 05:40:37,55 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower771: support() called 05:40:37,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - follower771: Converted InstallSnapshot from leader: leader769 to state 05:40:37,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - follower771: lastSequenceNumber prior to persisting applied snapshot: 3 05:40:37,56 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower771: support() called 05:40:37,56 PM [follower771-%d165372] [DEBUG] RaftStorage - follower771: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower771/snapshot-0000000068641D95-035BDD61.tmp 05:40:37,56 PM [leader769-%d328631] [DEBUG] RaftStorage - leader769: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D95-03416D49.v1 05:40:37,57 PM [leader769-%d328631] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-2CCAFE8C.v1 05:40:37,57 PM [leader769-%d328631] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D95-03416D49.v1 05:40:37,57 PM [leader769-%d328631] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/TermInfo.properties 05:40:37,57 PM [leader769-%d328631] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/journal-v1-1.log 05:40:37,57 PM [leader769-%d328631] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/metadata-v1-1.log 05:40:37,57 PM [leader769-%d328631] [DEBUG] RaftStorage - leader769: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-07-01T17:40:36.751500940Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D94-2CCAFE8C.v1, size=578, entryCompress=NONE, stateCompress=NONE} 05:40:37,57 PM [leader769-%d328631] [DEBUG] RaftStorage - leader769: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=8, term=6], timestamp=2025-07-01T17:40:37.054619465Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D95-03416D49.v1, size=500, entryCompress=NONE, stateCompress=NONE} 05:40:37,57 PM [leader769-%d328631] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:37,57 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:37,57 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader769: snapshot is durable as of 2025-07-01T17:40:37.054619465Z 05:40:37,57 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader769: Snapshot success - sequence number: 10 05:40:37,57 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorSnapshotMessageSupport - leader769: SnapshotComplete received 05:40:37,57 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 96.38 ms 05:40:37,58 PM [leader769-writer-163] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=10, applyTo=9] 05:40:37,58 PM [leader769-writer-163] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 05:40:37,58 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 05:40:37,58 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 206.3 μs 05:40:37,58 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 05:40:37,59 PM [follower771-%d165372] [DEBUG] RaftStorage - follower771: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower771/snapshot-0000000068641D95-035BDD61.v1 05:40:37,59 PM [follower771-%d165372] [DEBUG] RaftStorage - follower771: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower771/snapshot-0000000068641D95-035BDD61.v1 05:40:37,59 PM [follower771-%d165372] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower771/TermInfo.properties 05:40:37,59 PM [follower771-%d165372] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower771/journal-v1-1.log 05:40:37,59 PM [follower771-%d165372] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower771/metadata-v1-1.log 05:40:37,59 PM [follower771-%d165372] [DEBUG] RaftStorage - follower771: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=8, term=6], timestamp=2025-07-01T17:40:37.056352097Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/follower771/snapshot-0000000068641D95-035BDD61.v1, size=500, entryCompress=NONE, stateCompress=NONE} 05:40:37,59 PM [follower771-%d165372] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 05:40:37,59 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 05:40:37,59 PM [test-pekko.actor.default-dispatcher-15] [INFO] SnapshotManager - follower771: snapshot is durable as of 2025-07-01T17:40:37.056352097Z 05:40:37,59 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - follower771: Snapshot success - sequence number: 3 05:40:37,59 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower771: Updated server config: isVoting: false, peers: [PeerInfo [id=leader769, address=pekko://test/user/leader769, votingState=VOTING], PeerInfo [id=follower770, address=pekko://test/user/follower770, votingState=NON_VOTING]] 05:40:37,59 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower771: support() called 05:40:37,60 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower771: applySnapshot called 05:40:37,60 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=6, followerId=follower771, chunkIndex=1, success=true} 05:40:37,60 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader769 (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=6, followerId=follower771, chunkIndex=1, success=true} 05:40:37,60 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorSnapshotMessageSupport - follower771: SnapshotComplete received 05:40:37,60 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: received 1 commands after 1.063 s 05:40:37,60 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractLeader - leader769 (Leader): Snapshot successfully installed on follower follower771 (last chunk 1) - matchIndex set to 8, nextIndex set to 9 05:40:37,60 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader769: fakeSnapshot purging log to 7 for term 6 05:40:37,60 PM [follower771-writer-162] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=3, applyTo=2] 05:40:37,60 PM [follower771-writer-162] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 05:40:37,60 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 05:40:37,60 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 185.0 μs 05:40:37,60 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 05:40:37,173 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 8 (of expected -1) 05:40:37,173 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 8 (of expected -1) 05:40:37,173 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - follower770: fakeSnapshot purging log to 7 for term 6 05:40:37,293 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 8 (of expected -1) 05:40:37,293 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 8 (of expected -1) 05:40:37,307 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D95-03416D49.v1 05:40:37,307 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/TermInfo.properties 05:40:37,307 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/journal-v1-1.log 05:40:37,307 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/metadata-v1-1.log 05:40:37,307 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: picked SnapshotFileV1{lastIncluded=EntryInfo[index=8, term=6], timestamp=2025-07-01T17:40:37.054619465Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest4242417254831008791/leader769/snapshot-0000000068641D95-03416D49.v1, size=500, entryCompress=NONE, stateCompress=NONE} as the latest file 05:40:37,413 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 8 (of expected -1) 05:40:37,413 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 8 (of expected -1) 05:40:37,460 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - verifyInstallSnapshotToLaggingFollower complete 05:40:37,460 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaInstallSnapshot complete 05:40:37,460 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower770-collector772#-1762444269] 05:40:37,471 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower770] 05:40:37,471 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower770: terminating thread VirtualThread[#925,follower770-writer-161]/waiting 05:40:37,471 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: received 1 commands after 509.8 ms 05:40:37,471 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 45.00 μs 05:40:37,471 PM [follower770-writer-161] [DEBUG] JournalWriteTask - follower770: journal writer stopped 05:40:37,471 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower770: journal closed 05:40:37,472 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower770: journal closed 05:40:37,472 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower770: stopped executor with 0 remaining tasks 05:40:37,472 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower770: stopped executor with 0 remaining tasks 05:40:37,472 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower771-collector773#-1062037255] 05:40:37,472 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower771] 05:40:37,472 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower771: terminating thread VirtualThread[#926,follower771-writer-162]/waiting 05:40:37,472 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: received 1 commands after 411.8 ms 05:40:37,472 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 11.63 μs 05:40:37,472 PM [follower771-writer-162] [DEBUG] JournalWriteTask - follower771: journal writer stopped 05:40:37,472 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower771: journal closed 05:40:37,472 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower771: journal closed 05:40:37,472 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower771: stopped executor with 0 remaining tasks 05:40:37,472 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower771: stopped executor with 0 remaining tasks 05:40:37,474 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader769-collector774#1718185160] 05:40:37,474 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader769] 05:40:37,474 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - leader769: terminating thread VirtualThread[#927,leader769-writer-163]/waiting 05:40:37,474 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: received 1 commands after 416.7 ms 05:40:37,474 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 11.39 μs 05:40:37,474 PM [leader769-writer-163] [DEBUG] JournalWriteTask - leader769: journal writer stopped 05:40:37,475 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader769: journal closed 05:40:37,475 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader769: journal closed 05:40:37,475 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: stopped executor with 0 remaining tasks 05:40:37,475 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: stopped executor with 0 remaining tasks 05:40:37,477 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaAppendEntries starting 05:40:37,479 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower779: Starting recovery with journal batch size 1000 05:40:37,479 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower779: started executor 05:40:37,479 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower779: started executor [WARN] [07/01/2025 17:40:37.479] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower779]: org.opendaylight.controller.cluster.common.actor.Monitor@463cbad5 05:40:37,480 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower779: metadata journal open: replayFrom=1 applyTo=0 05:40:37,480 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower779: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:37,480 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower779: journal open: applyTo=0 05:40:37,481 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: journal writer started 05:40:37,481 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:37,482 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/follower779/TermInfo.properties does not exist 05:40:37,482 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower779: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/follower779/journal-v1-1.log 05:40:37,482 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower779: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/follower779/metadata-v1-1.log 05:40:37,482 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower779: no eligible files found 05:40:37,482 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower779: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/01/2025 17:40:37.483] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower780]: org.opendaylight.controller.cluster.common.actor.Monitor@5f371030 05:40:37,484 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower779: Set currentTerm=0, votedFor=null 05:40:37,484 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower779: Local TermInfo store seeded with TermInfo{term=0} 05:40:37,484 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower779: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@5a03643a 05:40:37,484 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower779: Moving last applied index from -1 to -1 05:40:37,484 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower779: empty journal: appending 0 entries 05:40:37,484 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower780: Starting recovery with journal batch size 1000 05:40:37,484 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower779: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:37,484 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower779: journal recovery completedin 228.6 μs with journalIndex=1 05:40:37,484 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower780: started executor 05:40:37,484 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower780: started executor 05:40:37,485 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower780: metadata journal open: replayFrom=1 applyTo=0 05:40:37,486 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower780: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:37,486 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower780: journal open: applyTo=0 05:40:37,486 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: journal writer started 05:40:37,486 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: waiting to receive commands 05:40:37,487 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/follower780/TermInfo.properties does not exist 05:40:37,487 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower780: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/follower780/journal-v1-1.log 05:40:37,487 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower780: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/follower780/metadata-v1-1.log 05:40:37,487 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower780: no eligible files found 05:40:37,487 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower780: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/01/2025 17:40:37.487] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader778]: org.opendaylight.controller.cluster.common.actor.Monitor@44ca5c36 05:40:37,488 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader778: Starting recovery with journal batch size 1000 05:40:37,488 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader778: started executor 05:40:37,489 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader778: started executor 05:40:37,489 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader778: metadata journal open: replayFrom=1 applyTo=0 05:40:37,489 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader778: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:37,489 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - leader778: journal open: applyTo=0 05:40:37,489 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: journal writer started 05:40:37,489 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:37,490 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/TermInfo.properties does not exist 05:40:37,490 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/journal-v1-1.log 05:40:37,490 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/metadata-v1-1.log 05:40:37,491 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader778: no eligible files found 05:40:37,491 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=5, votedFor=leader778 05:40:37,491 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader778: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:37,491 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower780: Set currentTerm=0, votedFor=null 05:40:37,491 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower780: Local TermInfo store seeded with TermInfo{term=0} 05:40:37,491 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower780: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@75dc0af 05:40:37,491 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower780: Moving last applied index from -1 to -1 05:40:37,491 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower780: empty journal: appending 0 entries 05:40:37,491 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower780: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:37,491 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower780: journal recovery completedin 218.1 μs with journalIndex=1 05:40:37,492 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=5, votedFor=leader778 05:40:37,493 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader778: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader778} 05:40:37,493 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@366a37cd 05:40:37,493 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader778: Moving last applied index from -1 to -1 05:40:37,493 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - leader778: empty journal: appending 0 entries 05:40:37,493 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader778: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:37,493 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - leader778: journal recovery completedin 207.1 μs with journalIndex=1 05:40:38,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 05:40:38,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - leader778 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.164:17355, Removed) 05:40:38,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - leader778 (Follower): Received ElectionTimeout - switching to Candidate 05:40:38,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader778 (Candidate): Election: Candidate has following voting peers: [follower779, follower780] 05:40:38,8 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=6, votedFor=leader778 05:40:38,8 PM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader778 (Candidate): Starting new election term 6 05:40:38,8 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader778 (Candidate): Sending RequestVote{term=6, candidateId=leader778, lastLogIndex=-1, lastLogTerm=-1} to peer follower779 05:40:38,8 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader778 (Candidate): Sending RequestVote{term=6, candidateId=leader778, lastLogIndex=-1, lastLogTerm=-1} to peer follower780 05:40:38,8 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - leader778 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 05:40:38,8 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower779 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 05:40:38,8 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower779 (Follower): Cluster state: [] 05:40:38,8 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower779 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader778, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 05:40:38,8 PM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower779 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader778, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 05:40:38,8 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower780 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 05:40:38,8 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower780 (Follower): Cluster state: [] 05:40:38,8 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower780 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader778, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 05:40:38,8 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower780 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader778, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 05:40:38,11 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower779: Set currentTerm=6, votedFor=null 05:40:38,11 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower779 (Follower): In requestVote: RequestVote{term=6, candidateId=leader778, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 05:40:38,11 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower780: Set currentTerm=6, votedFor=null 05:40:38,12 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower780 (Follower): In requestVote: RequestVote{term=6, candidateId=leader778, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 05:40:38,13 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower779: Set currentTerm=6, votedFor=leader778 05:40:38,13 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower779 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 05:40:38,14 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader778 (Candidate): RaftRPC message received RequestVoteReply{term=6, voteGranted=true}, my term is 6 05:40:38,14 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader778 (Candidate): handleRequestVoteReply: RequestVoteReply{term=6, voteGranted=true}, current voteCount: 1 05:40:38,14 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader778 (Leader): Election: Leader has following peers: [follower779, follower780] 05:40:38,14 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:38,14 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:38,14 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader778 (Candidate) :- Switching from behavior Candidate to Leader, election term: 6 05:40:38,14 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower779: Last sync leader does not match current leader leader778, need to catch up to -1 05:40:38,14 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower779 updated: matchIndex: -1, nextIndex: 0 05:40:38,14 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 17:40:38.014] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower779] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower779, initialSyncDone=false] 05:40:38,14 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower780: Set currentTerm=6, votedFor=leader778 05:40:38,14 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower780 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 05:40:38,15 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower780: Last sync leader does not match current leader leader778, need to catch up to -1 05:40:38,15 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower780 updated: matchIndex: -1, nextIndex: 0 05:40:38,15 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 17:40:38.015] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower780] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower780, initialSyncDone=false] 05:40:38,47 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - Leader created and elected 05:40:38,47 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Persist data index=0 term=6 command=zero 05:40:38,47 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader778: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:38,47 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 05:40:38,47 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 557.6 ms 05:40:38,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:38,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 0 is present for follower follower779 05:40:38,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower779: AppendEntries{term=6, leaderId=leader778, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:38,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:38,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 0 is present for follower follower780 05:40:38,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower780: AppendEntries{term=6, leaderId=leader778, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:38,48 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: awaiting more completions to resolve 1 synchronized callback(s) 05:40:38,48 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader778, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:38,48 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): Number of entries to be appended = 1 05:40:38,48 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 05:40:38,48 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): Append entry to log zero 05:40:38,48 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower779: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:38,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower780 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader778, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:38,48 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): Log size is now 1 05:40:38,48 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower779, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:38,48 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached -1 (of expected -1) 05:40:38,48 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 567.5 ms [WARN] [07/01/2025 17:40:38.048] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower779] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower779, initialSyncDone=true] 05:40:38,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower780 (Follower): Number of entries to be appended = 1 05:40:38,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower780 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 05:40:38,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower780 (Follower): Append entry to log zero 05:40:38,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower780: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:38,49 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower780 (Follower): Log size is now 1 05:40:38,49 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: received 1 commands after 562.9 ms 05:40:38,49 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower780 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower780, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:38,49 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower780: Lagging 0 entries behind leader leader778 and reached -1 (of expected -1) [WARN] [07/01/2025 17:40:38.049] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower780] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower780, initialSyncDone=true] 05:40:38,52 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,52 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 5.072 ms 05:40:38,52 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader778: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:38,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@71a7c8ac 05:40:38,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower779 updated: matchIndex: 0, nextIndex: 1 05:40:38,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 05:40:38,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader778 (Leader): Setting last applied to 0 05:40:38,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader778: Moving last applied index from -1 to 0 05:40:38,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Applying state for log index 0 data zero 05:40:38,52 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader778: applyState called: zero 05:40:38,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:38,53 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 511.4 μs 05:40:38,53 PM [leader778-writer-166] [DEBUG] EntryJournalV1 - leader778: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:38,53 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower780 updated: matchIndex: 0, nextIndex: 1 05:40:38,53 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 164.7 μs 05:40:38,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Commit index set to 0 05:40:38,53 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:38,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 0 (of expected -1) 05:40:38,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 05:40:38,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower779 (Follower): Setting last applied to 0 05:40:38,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower779: Moving last applied index from -1 to 0 05:40:38,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower779: Applying state for log index 0 data zero 05:40:38,53 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower779: applyState called: zero 05:40:38,53 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Commit index set to 0 05:40:38,53 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower780: Lagging 0 entries behind leader leader778 and reached 0 (of expected -1) 05:40:38,53 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 05:40:38,53 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower780 (Follower): Setting last applied to 0 05:40:38,53 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower780: Moving last applied index from -1 to 0 05:40:38,53 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower780: Applying state for log index 0 data zero 05:40:38,53 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower780: applyState called: zero 05:40:38,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Persist data index=1 term=6 command=one 05:40:38,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader778: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:38,54 PM [follower780-writer-165] [DEBUG] RaftStorageCompleter - follower780: 1 completion(s) pending 05:40:38,54 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 05:40:38,54 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: completed 1 commands in 5.083 ms 05:40:38,54 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,54 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 5.617 ms 05:40:38,54 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: received 1 commands after 3.720 μs 05:40:38,54 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 3.911 μs 05:40:38,54 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower779: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:38,54 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower780: running 1 completion(s) 05:40:38,54 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower780: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:38,54 PM [follower780-writer-165] [DEBUG] EntryJournalV1 - follower780: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:38,54 PM [follower780-writer-165] [DEBUG] RaftStorageCompleter - follower780: 1 completion(s) pending 05:40:38,54 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: completed 1 commands in 129.9 μs 05:40:38,54 PM [follower779-writer-164] [DEBUG] EntryJournalV1 - follower779: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:38,54 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 1.037 ms 05:40:38,54 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: waiting to receive commands 05:40:38,54 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,54 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 174.1 μs 05:40:38,54 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower780: running 1 completion(s) 05:40:38,54 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,54 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:38,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 1 is present for follower follower779 05:40:38,54 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower779: AppendEntries{term=6, leaderId=leader778, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:38,54 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader778, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:38,54 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): Number of entries to be appended = 1 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): Append entry to log one 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower779: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): Log size is now 2 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower779 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower779, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:38,55 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 543.1 μs 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 0 (of expected -1) 05:40:38,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:38,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 1 is present for follower follower780 05:40:38,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower780: AppendEntries{term=6, leaderId=leader778, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader778, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Number of entries to be appended = 1 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Append entry to log one 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower780: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Log size is now 2 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower780, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:38,55 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: received 1 commands after 1.091 ms 05:40:38,55 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower780: Lagging 0 entries behind leader leader778 and reached 0 (of expected -1) 05:40:38,55 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: awaiting more completions to resolve 1 synchronized callback(s) 05:40:38,56 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,56 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 1.797 ms 05:40:38,56 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader778: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:38,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@6401b987 05:40:38,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower779 updated: matchIndex: 1, nextIndex: 2 05:40:38,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 05:40:38,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader778 (Leader): Setting last applied to 1 05:40:38,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader778: Moving last applied index from 0 to 1 05:40:38,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Applying state for log index 1 data one 05:40:38,56 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader778: applyState called: one 05:40:38,56 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 393.1 μs 05:40:38,56 PM [leader778-writer-166] [DEBUG] EntryJournalV1 - leader778: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:38,56 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,56 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 131.3 μs 05:40:38,56 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,56 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - leader778: fakeSnapshot purging log to 0 for term 6 05:40:38,57 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 05:40:38,57 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,57 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower780 updated: matchIndex: 1, nextIndex: 2 05:40:38,57 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 05:40:38,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower779 (Follower): Commit index set to 1 05:40:38,57 PM [follower780-writer-165] [DEBUG] RaftStorageCompleter - follower780: 1 completion(s) pending 05:40:38,57 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 1 (of expected -1) 05:40:38,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower779 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 05:40:38,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower779 (Follower): Setting last applied to 1 05:40:38,57 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: completed 1 commands in 1.624 ms 05:40:38,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower779: Moving last applied index from 0 to 1 05:40:38,57 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: waiting to receive commands 05:40:38,57 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 2.126 ms 05:40:38,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower779: Applying state for log index 1 data one 05:40:38,57 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,57 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower779: applyState called: one 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower780: running 1 completion(s) 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower780: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 05:40:38,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower779: fakeSnapshot purging log to 0 for term 6 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower780 (Follower): Commit index set to 1 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower780: Lagging 0 entries behind leader leader778 and reached 1 (of expected -1) 05:40:38,57 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 154.0 μs 05:40:38,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower780 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower780 (Follower): Setting last applied to 1 05:40:38,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower779: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower780: Moving last applied index from 0 to 1 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower780: Applying state for log index 1 data one 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower780: applyState called: one 05:40:38,57 PM [follower779-writer-164] [DEBUG] EntryJournalV1 - follower779: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:38,57 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower780: fakeSnapshot purging log to 0 for term 6 05:40:38,57 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 154.0 μs 05:40:38,57 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,57 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: received 1 commands after 388.2 μs 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,57 PM [follower780-writer-165] [DEBUG] EntryJournalV1 - follower780: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:38,57 PM [follower780-writer-165] [DEBUG] RaftStorageCompleter - follower780: 1 completion(s) pending 05:40:38,57 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: completed 1 commands in 152.8 μs 05:40:38,57 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: waiting to receive commands 05:40:38,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower780: running 1 completion(s) 05:40:38,252 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 1 (of expected -1) 05:40:38,252 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower780: Lagging 0 entries behind leader leader778 and reached 1 (of expected -1) 05:40:38,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader778: Persist data index=2 term=6 command=two 05:40:38,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader778: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 05:40:38,299 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 242.4 ms 05:40:38,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): Replicate message: identifier: MockIdentifier{value=two}, logIndex: 2, isSendImmediate: true 05:40:38,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 05:40:38,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 2 is present for follower follower779 05:40:38,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower779: AppendEntries{term=6, leaderId=leader778, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 05:40:38,300 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 05:40:38,300 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 2 is present for follower follower780 05:40:38,300 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower780: AppendEntries{term=6, leaderId=leader778, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 05:40:38,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower779 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader778, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 05:40:38,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower779 (Follower): Number of entries to be appended = 1 05:40:38,300 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader778: awaiting more completions to resolve 1 synchronized callback(s) 05:40:38,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower779 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 05:40:38,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower779 (Follower): Append entry to log two 05:40:38,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower779: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 05:40:38,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower779 (Follower): Log size is now 2 05:40:38,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower779 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower779, logLastIndex=2, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:38,300 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 242.8 ms 05:40:38,300 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 1 (of expected -1) 05:40:38,301 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,301 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 2.033 ms 05:40:38,301 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,301 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,301 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader778: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=true} 05:40:38,301 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader778: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@18657863 05:40:38,301 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower779 updated: matchIndex: 2, nextIndex: 3 05:40:38,301 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 05:40:38,301 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader778 (Leader): Setting last applied to 2 05:40:38,301 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader778: Moving last applied index from 1 to 2 05:40:38,301 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader778: Applying state for log index 2 data two 05:40:38,301 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader778: applyState called: two 05:40:38,302 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 529.6 μs 05:40:38,302 PM [leader778-writer-166] [DEBUG] EntryJournalV1 - leader778: updated meta to JournalMeta[replayFrom=1, applyTo=3] 05:40:38,302 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,302 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 247.9 μs 05:40:38,302 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,302 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,302 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 1.948 ms 05:40:38,302 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,302 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader778: fakeSnapshot purging log to 1 for term 6 05:40:38,302 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,302 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower779: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=false} 05:40:38,302 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 1 05:40:38,302 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,302 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Commit index set to 2 05:40:38,302 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 2 (of expected -1) 05:40:38,302 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 2 05:40:38,302 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower779 (Follower): Setting last applied to 2 05:40:38,302 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower779: Moving last applied index from 1 to 2 05:40:38,302 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower779: Applying state for log index 2 data two 05:40:38,302 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower779: applyState called: two 05:40:38,303 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 572.3 μs 05:40:38,303 PM [follower779-writer-164] [DEBUG] EntryJournalV1 - follower779: updated meta to JournalMeta[replayFrom=1, applyTo=3] 05:40:38,303 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,303 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 135.4 μs 05:40:38,303 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,303 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - follower779: fakeSnapshot purging log to 1 for term 6 05:40:38,303 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,349 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Persist data index=3 term=6 command=three 05:40:38,349 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader778: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 05:40:38,349 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 47.41 ms 05:40:38,349 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Replicate message: identifier: MockIdentifier{value=three}, logIndex: 3, isSendImmediate: true 05:40:38,350 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 3, leaderLastIndex: 3, leaderSnapShotIndex: 1 05:40:38,350 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 3 is present for follower follower779 05:40:38,350 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower779: AppendEntries{term=6, leaderId=leader778, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 05:40:38,350 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: true, followerNextIndex: 2, leaderLastIndex: 3, leaderSnapShotIndex: 1 05:40:38,350 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 2 is present for follower follower780 05:40:38,350 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower780: AppendEntries{term=6, leaderId=leader778, prevLogIndex=1, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}, JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 05:40:38,350 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader778, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 05:40:38,350 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: awaiting more completions to resolve 1 synchronized callback(s) 05:40:38,350 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Number of entries to be appended = 1 05:40:38,350 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 05:40:38,350 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Append entry to log three 05:40:38,350 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower779: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 05:40:38,350 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Log size is now 2 05:40:38,350 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower779, logLastIndex=3, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:38,350 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 2 (of expected -1) 05:40:38,350 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 47.42 ms 05:40:38,351 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,351 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 1.910 ms 05:40:38,351 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,351 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,351 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader778: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=true} 05:40:38,351 PM [test-pekko.actor.default-dispatcher-15] [INFO] SnapshotManager - leader778: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=2, lastAppliedTerm=6, lastIndex=3, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=1, mandatoryTrim=false] 05:40:38,351 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - leader778: captured snapshot at lastSequenceNumber: 4 05:40:38,351 PM [test-pekko.actor.default-dispatcher-15] [INFO] SnapshotManager - leader778: Persising snapshot at EntryInfo[index=2, term=6]/EntryInfo[index=3, term=6] 05:40:38,352 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader778: support() called 05:40:38,352 PM [leader778-%d60586] [DEBUG] RaftStorage - leader778: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/snapshot-0000000068641D96-15019E4C.tmp 05:40:38,352 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,352 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 2.174 ms 05:40:38,352 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,352 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,352 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower779: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=false} 05:40:38,353 PM [test-pekko.actor.default-dispatcher-15] [INFO] SnapshotManager - leader778: Removed in-memory snapshotted entries, adjusted snaphsotIndex: 1 and term: 6 05:40:38,353 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@3dcf55fa 05:40:38,353 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower779 updated: matchIndex: 3, nextIndex: 4 05:40:38,353 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2 05:40:38,353 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader778 (Leader): Setting last applied to 3 05:40:38,353 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader778: Moving last applied index from 2 to 3 05:40:38,353 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Applying state for log index 3 data three 05:40:38,353 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader778: applyState called: three 05:40:38,353 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 4, leaderLastIndex: 3, leaderSnapShotIndex: 1 05:40:38,353 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 1.904 ms 05:40:38,353 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower779 (Follower): Commit index set to 3 05:40:38,353 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 3 (of expected -1) 05:40:38,353 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower779 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): 2, lastIndex(): 3 05:40:38,353 PM [leader778-writer-166] [DEBUG] EntryJournalV1 - leader778: updated meta to JournalMeta[replayFrom=1, applyTo=4] 05:40:38,353 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower779 (Follower): Setting last applied to 3 05:40:38,353 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,353 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower779: Moving last applied index from 2 to 3 05:40:38,353 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower779: Applying state for log index 3 data three 05:40:38,353 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower779: applyState called: three 05:40:38,353 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 270.3 μs 05:40:38,353 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,354 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 1.137 ms 05:40:38,354 PM [follower779-writer-164] [DEBUG] EntryJournalV1 - follower779: updated meta to JournalMeta[replayFrom=1, applyTo=4] 05:40:38,354 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,354 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 119.6 μs 05:40:38,354 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,354 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,354 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,354 PM [leader778-%d60586] [DEBUG] RaftStorage - leader778: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/snapshot-0000000068641D96-15019E4C.v1 05:40:38,355 PM [leader778-%d60586] [DEBUG] RaftStorage - leader778: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/snapshot-0000000068641D96-15019E4C.v1 05:40:38,355 PM [leader778-%d60586] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/TermInfo.properties 05:40:38,355 PM [leader778-%d60586] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/journal-v1-1.log 05:40:38,355 PM [leader778-%d60586] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/metadata-v1-1.log 05:40:38,355 PM [leader778-%d60586] [DEBUG] RaftStorage - leader778: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-07-01T17:40:38.352427596Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/snapshot-0000000068641D96-15019E4C.v1, size=510, entryCompress=NONE, stateCompress=NONE} 05:40:38,355 PM [leader778-%d60586] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,355 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,355 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader778: snapshot is durable as of 2025-07-01T17:40:38.352427596Z 05:40:38,355 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader778: Snapshot success - sequence number: 4 05:40:38,355 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorSnapshotMessageSupport - leader778: SnapshotComplete received 05:40:38,355 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 1.687 ms 05:40:38,355 PM [leader778-writer-166] [DEBUG] EntryJournalV1 - leader778: updated meta to JournalMeta[replayFrom=4, applyTo=4] 05:40:38,355 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,355 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 198.7 μs 05:40:38,355 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,356 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,372 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 2 is present for follower follower780 05:40:38,372 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower780: AppendEntries{term=6, leaderId=leader778, prevLogIndex=1, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}, JournaledLogEntry{index=3, term=6, command=three, pending=false}]} 05:40:38,450 PM [awaitility-thread] [DEBUG] RaftStorage - leader778: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/snapshot-0000000068641D96-15019E4C.v1 05:40:38,450 PM [awaitility-thread] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/TermInfo.properties 05:40:38,450 PM [awaitility-thread] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/journal-v1-1.log 05:40:38,450 PM [awaitility-thread] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/metadata-v1-1.log 05:40:38,450 PM [awaitility-thread] [DEBUG] RaftStorage - leader778: picked SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-07-01T17:40:38.352427596Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6670029676296268622/leader778/snapshot-0000000068641D96-15019E4C.v1, size=510, entryCompress=NONE, stateCompress=NONE} as the latest file 05:40:38,451 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaAppendEntries: sending 2 more payloads 05:40:38,451 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Persist data index=4 term=6 command=four 05:40:38,451 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader778: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 05:40:38,451 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Replicate message: identifier: MockIdentifier{value=four}, logIndex: 4, isSendImmediate: true 05:40:38,451 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 4, leaderLastIndex: 4, leaderSnapShotIndex: 1 05:40:38,451 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 4 is present for follower follower779 05:40:38,451 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 95.38 ms 05:40:38,451 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower779: AppendEntries{term=6, leaderId=leader778, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 05:40:38,451 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: true, followerNextIndex: 2, leaderLastIndex: 4, leaderSnapShotIndex: 1 05:40:38,451 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 2 is present for follower follower780 05:40:38,451 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: awaiting more completions to resolve 1 synchronized callback(s) 05:40:38,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader778, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 05:40:38,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Number of entries to be appended = 1 05:40:38,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): After cleanup, lastIndex: 3, entries to be added from: 0 05:40:38,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Append entry to log four 05:40:38,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower779: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 05:40:38,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Log size is now 3 05:40:38,451 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 97.49 ms 05:40:38,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower779, logLastIndex=4, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:38,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 3 (of expected -1) 05:40:38,453 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,453 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,453 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 2.042 ms 05:40:38,453 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,453 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 1.619 ms 05:40:38,453 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,453 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,453 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower779: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=false} 05:40:38,453 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,453 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader778: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=true} 05:40:38,453 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@753adf50 05:40:38,453 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower779 updated: matchIndex: 4, nextIndex: 5 05:40:38,453 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Applying to log - commitIndex: 4, lastAppliedIndex: 3 05:40:38,453 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader778 (Leader): Setting last applied to 4 05:40:38,453 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader778: Moving last applied index from 3 to 4 05:40:38,453 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Applying state for log index 4 data four 05:40:38,453 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader778: applyState called: four 05:40:38,453 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 5, leaderLastIndex: 4, leaderSnapShotIndex: 1 05:40:38,453 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 434.2 μs 05:40:38,454 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower779 (Follower): Commit index set to 4 05:40:38,454 PM [leader778-writer-166] [DEBUG] EntryJournalV1 - leader778: updated meta to JournalMeta[replayFrom=4, applyTo=5] 05:40:38,454 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 4 (of expected -1) 05:40:38,454 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,454 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Persist data index=5 term=6 command=five 05:40:38,454 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower779 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 4, context.getLastApplied(): 3, lastIndex(): 4 05:40:38,454 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower779 (Follower): Setting last applied to 4 05:40:38,454 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader778: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 05:40:38,454 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower779: Moving last applied index from 3 to 4 05:40:38,454 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 147.4 μs 05:40:38,454 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,454 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower779: Applying state for log index 4 data four 05:40:38,454 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower779: applyState called: four 05:40:38,454 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Replicate message: identifier: MockIdentifier{value=five}, logIndex: 5, isSendImmediate: true 05:40:38,454 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 68.55 μs 05:40:38,454 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 5, leaderLastIndex: 5, leaderSnapShotIndex: 1 05:40:38,454 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 5 is present for follower follower779 05:40:38,454 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 795.9 μs 05:40:38,454 PM [follower779-writer-164] [DEBUG] EntryJournalV1 - follower779: updated meta to JournalMeta[replayFrom=1, applyTo=5] 05:40:38,454 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,454 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 179.0 μs 05:40:38,454 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,454 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower779: AppendEntries{term=6, leaderId=leader778, prevLogIndex=4, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 05:40:38,454 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: true, followerNextIndex: 2, leaderLastIndex: 5, leaderSnapShotIndex: 1 05:40:38,454 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 2 is present for follower follower780 05:40:38,454 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower780: AppendEntries{term=6, leaderId=leader778, prevLogIndex=1, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}, JournaledLogEntry{index=3, term=6, command=three, pending=false}, JournaledLogEntry{index=4, term=6, command=four, pending=false}, JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 05:40:38,454 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,454 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,455 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: awaiting more completions to resolve 1 synchronized callback(s) 05:40:38,455 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader778, prevLogIndex=4, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 05:40:38,455 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Number of entries to be appended = 1 05:40:38,455 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): After cleanup, lastIndex: 4, entries to be added from: 0 05:40:38,455 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Append entry to log five 05:40:38,455 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower779: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 05:40:38,455 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): Log size is now 4 05:40:38,455 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower779 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower779, logLastIndex=5, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:38,455 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 4 (of expected -1) 05:40:38,455 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 568.0 μs 05:40:38,455 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,455 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,455 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 1.717 ms 05:40:38,455 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,455 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader778: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=true} 05:40:38,456 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@36f8ef3d 05:40:38,456 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,456 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 837.4 μs 05:40:38,456 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,456 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,456 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower779: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=false} 05:40:38,456 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 5 is present for follower follower779 05:40:38,456 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower779 updated: matchIndex: 5, nextIndex: 6 05:40:38,456 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Applying to log - commitIndex: 5, lastAppliedIndex: 4 05:40:38,456 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader778 (Leader): Setting last applied to 5 05:40:38,456 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader778: Moving last applied index from 4 to 5 05:40:38,456 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader778: Applying state for log index 5 data five 05:40:38,456 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader778: applyState called: five 05:40:38,456 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 777.1 μs 05:40:38,456 PM [leader778-writer-166] [DEBUG] EntryJournalV1 - leader778: updated meta to JournalMeta[replayFrom=4, applyTo=6] 05:40:38,456 PM [leader778-writer-166] [DEBUG] RaftStorageCompleter - leader778: 1 completion(s) pending 05:40:38,456 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 156.7 μs 05:40:38,456 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 05:40:38,457 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower779: active: true, followerNextIndex: 6, leaderLastIndex: 5, leaderSnapShotIndex: 1 05:40:38,457 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader778: running 1 completion(s) 05:40:38,457 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower779 (Follower): Commit index set to 5 05:40:38,457 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 5 (of expected -1) 05:40:38,457 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower779 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 5, context.getLastApplied(): 4, lastIndex(): 5 05:40:38,457 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower779 (Follower): Setting last applied to 5 05:40:38,457 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower779: Moving last applied index from 4 to 5 05:40:38,457 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower779: Applying state for log index 5 data five 05:40:38,457 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower779: applyState called: five 05:40:38,457 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 1.271 ms 05:40:38,457 PM [follower779-writer-164] [DEBUG] EntryJournalV1 - follower779: updated meta to JournalMeta[replayFrom=1, applyTo=6] 05:40:38,457 PM [follower779-writer-164] [DEBUG] RaftStorageCompleter - follower779: 1 completion(s) pending 05:40:38,457 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 163.9 μs 05:40:38,457 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower779: running 1 completion(s) 05:40:38,457 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 05:40:38,492 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 2 is present for follower follower780 05:40:38,492 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower780: AppendEntries{term=6, leaderId=leader778, prevLogIndex=1, prevLogTerm=6, leaderCommit=5, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}, JournaledLogEntry{index=3, term=6, command=three, pending=false}, JournaledLogEntry{index=4, term=6, command=four, pending=false}, JournaledLogEntry{index=5, term=6, command=five, pending=false}]} 05:40:38,612 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader778 (Leader): sendAppendEntries: 2 is present for follower follower780 05:40:38,612 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader778 (Leader): Sending AppendEntries to follower follower780: AppendEntries{term=6, leaderId=leader778, prevLogIndex=1, prevLogTerm=6, leaderCommit=5, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}, JournaledLogEntry{index=3, term=6, command=three, pending=false}, JournaledLogEntry{index=4, term=6, command=four, pending=false}, JournaledLogEntry{index=5, term=6, command=five, pending=false}]} 05:40:38,612 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 5 (of expected -1) 05:40:38,612 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader778, prevLogIndex=1, prevLogTerm=6, leaderCommit=5, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}, JournaledLogEntry{index=3, term=6, command=three, pending=false}, JournaledLogEntry{index=4, term=6, command=four, pending=false}, JournaledLogEntry{index=5, term=6, command=five, pending=false}]} 05:40:38,612 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Number of entries to be appended = 4 05:40:38,612 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 05:40:38,612 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Append entry to log two 05:40:38,612 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower780: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=false} 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Append entry to log three 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower780: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=false} 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Append entry to log four 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower780: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=false} 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Append entry to log five 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower780: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=false} 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Log size is now 5 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): Commit index set to 5 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower780, logLastIndex=5, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower780: Lagging 0 entries behind leader leader778 and reached 5 (of expected -1) 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower780 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 5, context.getLastApplied(): 1, lastIndex(): 5 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower780 (Follower): Setting last applied to 2 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower780: Moving last applied index from 1 to 2 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower780: Applying state for log index 2 data two 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower780: applyState called: two 05:40:38,613 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader778 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower780 updated: matchIndex: 5, nextIndex: 6 05:40:38,613 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader778: fakeSnapshot purging log to 4 for term 6 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower780 (Follower): Setting last applied to 3 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower780: Moving last applied index from 2 to 3 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower780: Applying state for log index 3 data three 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower780: applyState called: three 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower780 (Follower): Setting last applied to 4 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower780: Moving last applied index from 3 to 4 05:40:38,613 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader778 (Leader): Checking sendAppendEntries for follower follower780: active: true, followerNextIndex: 6, leaderLastIndex: 5, leaderSnapShotIndex: 4 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower780: Applying state for log index 4 data four 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower780: applyState called: four 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower780 (Follower): Setting last applied to 5 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower780: Moving last applied index from 4 to 5 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower780: Applying state for log index 5 data five 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower780: applyState called: five 05:40:38,613 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: received 4 commands after 555.7 ms 05:40:38,613 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - follower780: fakeSnapshot purging log to 1 for term 6 05:40:38,618 PM [follower780-writer-165] [DEBUG] RaftStorageCompleter - follower780: 4 completion(s) pending 05:40:38,618 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: completed 4 commands in 4.847 ms 05:40:38,618 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: received 1 commands after 4.050 μs 05:40:38,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower780: running 4 completion(s) 05:40:38,618 PM [follower780-writer-165] [DEBUG] EntryJournalV1 - follower780: updated meta to JournalMeta[replayFrom=1, applyTo=6] 05:40:38,618 PM [follower780-writer-165] [DEBUG] RaftStorageCompleter - follower780: 1 completion(s) pending 05:40:38,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower780: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=false} 05:40:38,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower780: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=false} 05:40:38,618 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: completed 1 commands in 157.2 μs 05:40:38,618 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: waiting to receive commands 05:40:38,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower780: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=false} 05:40:38,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower780: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=false} 05:40:38,618 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower780: running 1 completion(s) 05:40:38,732 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 5 (of expected -1) 05:40:38,732 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower780: Lagging 0 entries behind leader leader778 and reached 5 (of expected -1) 05:40:38,733 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - follower780: fakeSnapshot purging log to 4 for term 6 05:40:38,733 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - follower779: fakeSnapshot purging log to 4 for term 6 05:40:38,852 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 5 (of expected -1) 05:40:38,852 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower780: Lagging 0 entries behind leader leader778 and reached 5 (of expected -1) 05:40:38,972 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower779: Lagging 0 entries behind leader leader778 and reached 5 (of expected -1) 05:40:38,973 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower780: Lagging 0 entries behind leader leader778 and reached 5 (of expected -1) 05:40:39,5 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaAppendEntries complete 05:40:39,5 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower779-collector781#1449968270] 05:40:39,6 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower779] 05:40:39,6 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower779: terminating thread VirtualThread[#938,follower779-writer-164]/waiting 05:40:39,6 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: received 1 commands after 549.2 ms 05:40:39,6 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 15.17 μs 05:40:39,6 PM [follower779-writer-164] [DEBUG] JournalWriteTask - follower779: journal writer stopped 05:40:39,7 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower779: journal closed 05:40:39,7 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower779: journal closed 05:40:39,7 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower779: stopped executor with 0 remaining tasks 05:40:39,7 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower779: stopped executor with 0 remaining tasks 05:40:39,7 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower780-collector782#1298341516] 05:40:39,7 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower780] 05:40:39,7 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower780: terminating thread VirtualThread[#939,follower780-writer-165]/waiting 05:40:39,7 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: received 1 commands after 389.0 ms 05:40:39,7 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: completed 1 commands in 9.660 μs 05:40:39,7 PM [follower780-writer-165] [DEBUG] JournalWriteTask - follower780: journal writer stopped 05:40:39,7 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower780: journal closed 05:40:39,7 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower780: journal closed 05:40:39,7 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower780: stopped executor with 0 remaining tasks 05:40:39,7 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower780: stopped executor with 0 remaining tasks 05:40:39,8 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader778-collector783#-1609464882] 05:40:39,8 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader778] 05:40:39,8 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - leader778: terminating thread VirtualThread[#940,leader778-writer-166]/waiting 05:40:39,8 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: received 1 commands after 551.5 ms 05:40:39,8 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 9.890 μs 05:40:39,8 PM [leader778-writer-166] [DEBUG] JournalWriteTask - leader778: journal writer stopped 05:40:39,9 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader778: journal closed 05:40:39,9 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - leader778: journal closed 05:40:39,9 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader778: stopped executor with 0 remaining tasks 05:40:39,9 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader778: stopped executor with 0 remaining tasks 05:40:39,11 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderInstallsSnapshotWithRestartedFollowerDuringSnapshotInstallation starting [WARN] [07/01/2025 17:40:39.012] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower788]: org.opendaylight.controller.cluster.common.actor.Monitor@78366cff 05:40:39,13 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower788: Starting recovery with journal batch size 1000 05:40:39,13 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower788: started executor 05:40:39,13 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower788: started executor 05:40:39,13 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower788: metadata journal open: replayFrom=1 applyTo=0 05:40:39,14 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower788: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:39,14 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower788: journal open: applyTo=0 05:40:39,14 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: journal writer started 05:40:39,14 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:39,15 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower788/TermInfo.properties does not exist 05:40:39,15 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower788: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower788/journal-v1-1.log 05:40:39,15 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower788: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower788/metadata-v1-1.log 05:40:39,15 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower788: no eligible files found 05:40:39,15 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower788: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/01/2025 17:40:39.016] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower789]: org.opendaylight.controller.cluster.common.actor.Monitor@374eeb74 05:40:39,16 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower789: Starting recovery with journal batch size 1000 05:40:39,17 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: started executor 05:40:39,17 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: started executor 05:40:39,17 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower789: metadata journal open: replayFrom=1 applyTo=0 05:40:39,18 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower789: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:39,18 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower789: journal open: applyTo=0 05:40:39,18 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower788: Set currentTerm=0, votedFor=null 05:40:39,18 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower788: Local TermInfo store seeded with TermInfo{term=0} 05:40:39,18 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower788: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@5fed6984 05:40:39,18 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower788: Moving last applied index from -1 to -1 05:40:39,18 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: journal writer started 05:40:39,18 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: waiting to receive commands 05:40:39,18 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower788: empty journal: appending 0 entries 05:40:39,18 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower788: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:39,18 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower788: journal recovery completedin 248.5 μs with journalIndex=1 05:40:39,19 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/TermInfo.properties does not exist [WARN] [07/01/2025 17:40:39.019] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader787]: org.opendaylight.controller.cluster.common.actor.Monitor@30ba7105 05:40:39,19 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/journal-v1-1.log 05:40:39,19 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/metadata-v1-1.log 05:40:39,19 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: no eligible files found 05:40:39,19 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower789: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:39,20 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader787: Starting recovery with journal batch size 1000 05:40:39,20 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader787: started executor 05:40:39,20 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader787: started executor 05:40:39,20 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader787: metadata journal open: replayFrom=1 applyTo=0 05:40:39,21 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader787: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:39,21 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - leader787: journal open: applyTo=0 05:40:39,21 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: journal writer started 05:40:39,21 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:39,22 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/TermInfo.properties does not exist 05:40:39,22 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/journal-v1-1.log 05:40:39,22 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/metadata-v1-1.log 05:40:39,22 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader787: no eligible files found 05:40:39,22 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=5, votedFor=leader787 05:40:39,22 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader787: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:39,23 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower789: Set currentTerm=0, votedFor=null 05:40:39,23 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower789: Local TermInfo store seeded with TermInfo{term=0} 05:40:39,23 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower789: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@7ca095c6 05:40:39,23 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower789: Moving last applied index from -1 to -1 05:40:39,23 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower789: empty journal: appending 0 entries 05:40:39,23 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower789: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:39,23 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower789: journal recovery completedin 491.5 μs with journalIndex=1 05:40:39,24 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=5, votedFor=leader787 05:40:39,24 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader787: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader787} 05:40:39,24 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader787: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6b64882d 05:40:39,24 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader787: Moving last applied index from -1 to -1 05:40:39,25 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader787: empty journal: appending 0 entries 05:40:39,25 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:39,25 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader787: journal recovery completedin 152.9 μs with journalIndex=1 05:40:39,482 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 05:40:39,482 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader787 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.164:17355, Removed) 05:40:39,482 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader787 (Follower): Received ElectionTimeout - switching to Candidate 05:40:39,482 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader787 (Candidate): Election: Candidate has following voting peers: [follower788, follower789] 05:40:39,487 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=6, votedFor=leader787 05:40:39,487 PM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader787 (Candidate): Starting new election term 6 05:40:39,487 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader787 (Candidate): Sending RequestVote{term=6, candidateId=leader787, lastLogIndex=-1, lastLogTerm=-1} to peer follower788 05:40:39,487 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader787 (Candidate): Sending RequestVote{term=6, candidateId=leader787, lastLogIndex=-1, lastLogTerm=-1} to peer follower789 05:40:39,488 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower789 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 05:40:39,488 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader787 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 05:40:39,488 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower789 (Follower): Cluster state: [] 05:40:39,488 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower789 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader787, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 05:40:39,488 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower789 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader787, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 05:40:39,488 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower788 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 05:40:39,488 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower788 (Follower): Cluster state: [] 05:40:39,488 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower788 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader787, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 05:40:39,488 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower788 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader787, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 05:40:39,490 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower789: Set currentTerm=6, votedFor=null 05:40:39,490 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower789 (Follower): In requestVote: RequestVote{term=6, candidateId=leader787, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 05:40:39,492 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower788: Set currentTerm=6, votedFor=null 05:40:39,492 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower788 (Follower): In requestVote: RequestVote{term=6, candidateId=leader787, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 05:40:39,493 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower789: Set currentTerm=6, votedFor=leader787 05:40:39,493 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower789 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 05:40:39,493 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader787 (Candidate): RaftRPC message received RequestVoteReply{term=6, voteGranted=true}, my term is 6 05:40:39,493 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader787 (Candidate): handleRequestVoteReply: RequestVoteReply{term=6, voteGranted=true}, current voteCount: 1 05:40:39,493 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Election: Leader has following peers: [follower788, follower789] 05:40:39,494 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:39,494 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:39,494 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader787 (Candidate) :- Switching from behavior Candidate to Leader, election term: 6 05:40:39,494 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower789: Last sync leader does not match current leader leader787, need to catch up to -1 05:40:39,494 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower789 updated: matchIndex: -1, nextIndex: 0 05:40:39,494 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:39,494 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower788: Set currentTerm=6, votedFor=leader787 [WARN] [07/01/2025 17:40:39.494] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower789] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower789, initialSyncDone=false] 05:40:39,494 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower788 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 05:40:39,494 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower788: Last sync leader does not match current leader leader787, need to catch up to -1 05:40:39,494 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower788 updated: matchIndex: -1, nextIndex: 0 05:40:39,494 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 17:40:39.494] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower788] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower788, initialSyncDone=false] 05:40:39,528 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - Leader created and elected 05:40:39,528 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader787: Persist data index=0 term=6 command=zero 05:40:39,528 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader787: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:39,528 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 05:40:39,528 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:39,528 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 0 is present for follower follower788 05:40:39,528 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 506.9 ms 05:40:39,528 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Sending AppendEntries to follower follower788: AppendEntries{term=6, leaderId=leader787, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:39,528 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:39,528 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 0 is present for follower follower789 05:40:39,528 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Sending AppendEntries to follower follower789: AppendEntries{term=6, leaderId=leader787, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:39,528 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: awaiting more completions to resolve 1 synchronized callback(s) 05:40:39,528 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower788 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader787, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:39,528 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower789 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader787, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:39,528 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower789 (Follower): Number of entries to be appended = 1 05:40:39,528 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower788 (Follower): Number of entries to be appended = 1 05:40:39,528 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower789 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 05:40:39,528 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower788 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 05:40:39,528 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower789 (Follower): Append entry to log zero 05:40:39,528 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower789: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:39,528 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower788 (Follower): Append entry to log zero 05:40:39,528 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower788: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:39,528 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower789 (Follower): Log size is now 1 05:40:39,528 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower789 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower789, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:39,528 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower789: Lagging 0 entries behind leader leader787 and reached -1 (of expected -1) 05:40:39,528 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: received 1 commands after 510.6 ms 05:40:39,528 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower788 (Follower): Log size is now 1 05:40:39,528 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower788 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower788, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:39,528 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached -1 (of expected -1) [WARN] [07/01/2025 17:40:39.528] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower789] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower789, initialSyncDone=true] 05:40:39,528 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 514.7 ms [WARN] [07/01/2025 17:40:39.529] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower788] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower788, initialSyncDone=true] 05:40:39,532 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:39,532 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 4.319 ms 05:40:39,532 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:39,532 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:39,532 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader787: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:39,532 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@60da839b 05:40:39,532 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower789 updated: matchIndex: 0, nextIndex: 1 05:40:39,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 05:40:39,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader787 (Leader): Setting last applied to 0 05:40:39,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader787: Moving last applied index from -1 to 0 05:40:39,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader787: Applying state for log index 0 data zero 05:40:39,533 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader787: applyState called: zero 05:40:39,533 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 473.6 μs 05:40:39,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:39,533 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:39,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower788 updated: matchIndex: 0, nextIndex: 1 05:40:39,533 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:39,533 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower789 (Follower): Commit index set to 0 05:40:39,533 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 159.1 μs 05:40:39,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:39,533 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:39,533 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower789: Lagging 0 entries behind leader leader787 and reached 0 (of expected -1) 05:40:39,533 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower789 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 05:40:39,533 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower789 (Follower): Setting last applied to 0 05:40:39,533 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower789: Moving last applied index from -1 to 0 05:40:39,533 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): Commit index set to 0 05:40:39,533 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower789: Applying state for log index 0 data zero 05:40:39,533 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 0 (of expected -1) 05:40:39,533 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower789: applyState called: zero 05:40:39,533 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 05:40:39,533 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower788 (Follower): Setting last applied to 0 05:40:39,533 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower788: Moving last applied index from -1 to 0 05:40:39,533 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower788: Applying state for log index 0 data zero 05:40:39,533 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower788: applyState called: zero 05:40:39,533 PM [follower789-writer-168] [DEBUG] RaftStorageCompleter - follower789: 1 completion(s) pending 05:40:39,533 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:39,533 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: completed 1 commands in 4.966 ms 05:40:39,533 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:39,533 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 4.823 ms 05:40:39,533 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: waiting to receive commands 05:40:39,533 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower788: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:39,533 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 3.120 μs 05:40:39,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:39,533 PM [follower788-writer-167] [DEBUG] EntryJournalV1 - follower788: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:39,533 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:39,533 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: received 1 commands after 142.2 μs 05:40:39,534 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 112.2 μs 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:39,534 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:39,534 PM [follower789-writer-168] [DEBUG] EntryJournalV1 - follower789: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:39,534 PM [follower789-writer-168] [DEBUG] RaftStorageCompleter - follower789: 1 more completion(s) pending 05:40:39,534 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: completed 1 commands in 148.7 μs 05:40:39,534 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: waiting to receive commands 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader787: Persist data index=1 term=6 command=one 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader787: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:39,534 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower789: running 2 completion(s) 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader787 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 05:40:39,534 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 877.1 μs 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 1 is present for follower follower788 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader787 (Leader): Sending AppendEntries to follower follower788: AppendEntries{term=6, leaderId=leader787, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 1 is present for follower follower789 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader787 (Leader): Sending AppendEntries to follower follower789: AppendEntries{term=6, leaderId=leader787, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:39,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader787: awaiting more completions to resolve 1 synchronized callback(s) 05:40:39,534 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader787, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:39,534 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower789: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:39,534 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): Number of entries to be appended = 1 05:40:39,534 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 05:40:39,534 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): Append entry to log one 05:40:39,534 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower788: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:39,534 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): Log size is now 2 05:40:39,534 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower789 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader787, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:39,534 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 719.1 μs 05:40:39,534 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower789 (Follower): Number of entries to be appended = 1 05:40:39,534 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower788, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:39,534 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower789 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 05:40:39,534 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 0 (of expected -1) 05:40:39,534 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower789 (Follower): Append entry to log one 05:40:39,534 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower789: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:39,534 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower789 (Follower): Log size is now 2 05:40:39,534 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower789 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower789, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:39,535 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower789: Lagging 0 entries behind leader leader787 and reached 0 (of expected -1) 05:40:39,535 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: received 1 commands after 802.9 μs 05:40:39,535 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:39,535 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 1.559 ms 05:40:39,535 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:39,535 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader787: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader787: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@2b7e09a3 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower788 updated: matchIndex: 1, nextIndex: 2 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader787 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader787 (Leader): Setting last applied to 1 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader787: Moving last applied index from 0 to 1 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader787: Applying state for log index 1 data one 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader787: applyState called: one 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader787: fakeSnapshot purging log to 0 for term 6 05:40:39,536 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 412.6 μs 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 05:40:39,536 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:39,536 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:39,536 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 115.6 μs 05:40:39,536 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): Commit index set to 1 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 1 (of expected -1) 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 05:40:39,536 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower788 (Follower): Setting last applied to 1 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower788: Moving last applied index from 0 to 1 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower788: Applying state for log index 1 data one 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower788: applyState called: one 05:40:39,536 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower789 updated: matchIndex: 1, nextIndex: 2 05:40:39,536 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 05:40:39,536 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:39,536 PM [follower789-writer-168] [DEBUG] RaftStorageCompleter - follower789: 1 completion(s) pending 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower788: fakeSnapshot purging log to 0 for term 6 05:40:39,536 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 1.902 ms 05:40:39,536 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: completed 1 commands in 1.746 ms 05:40:39,536 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 3.700 μs 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:39,536 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: waiting to receive commands 05:40:39,536 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower789: running 1 completion(s) 05:40:39,536 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower788: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 05:40:39,536 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower789: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 05:40:39,536 PM [follower788-writer-167] [DEBUG] EntryJournalV1 - follower788: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:39,536 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:39,536 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower789 (Follower): Commit index set to 1 05:40:39,536 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower789: Lagging 0 entries behind leader leader787 and reached 1 (of expected -1) 05:40:39,536 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower789 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 05:40:39,536 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 129.7 μs 05:40:39,536 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower789 (Follower): Setting last applied to 1 05:40:39,536 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:39,536 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower789: Moving last applied index from 0 to 1 05:40:39,536 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:39,537 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower789: Applying state for log index 1 data one 05:40:39,537 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower789: applyState called: one 05:40:39,537 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - follower789: fakeSnapshot purging log to 0 for term 6 05:40:39,537 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: received 1 commands after 312.9 μs 05:40:39,537 PM [follower789-writer-168] [DEBUG] EntryJournalV1 - follower789: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:39,537 PM [follower789-writer-168] [DEBUG] RaftStorageCompleter - follower789: 1 completion(s) pending 05:40:39,537 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: completed 1 commands in 144.8 μs 05:40:39,537 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: waiting to receive commands 05:40:39,537 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower789: running 1 completion(s) 05:40:39,733 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower789: Lagging 0 entries behind leader leader787 and reached 1 (of expected -1) 05:40:39,733 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 1 (of expected -1) 05:40:39,779 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRaftStorage - follower789: terminating thread VirtualThread[#944,follower789-writer-168]/waiting 05:40:39,779 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: received 1 commands after 242.4 ms 05:40:39,779 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: completed 1 commands in 25.03 μs 05:40:39,779 PM [follower789-writer-168] [DEBUG] JournalWriteTask - follower789: journal writer stopped 05:40:39,780 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower789: journal closed 05:40:39,780 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - follower789: journal closed 05:40:39,780 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower789: stopped executor with 0 remaining tasks 05:40:39,780 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower789: stopped executor with 0 remaining tasks 05:40:39,853 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 1 (of expected -1) [WARN] [07/01/2025 17:40:39.853] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [07/01/2025 17:40:39.973] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 05:40:39,973 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 1 (of expected -1) [WARN] [07/01/2025 17:40:40.093] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 05:40:40,93 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 1 (of expected -1) 05:40:40,184 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader787: Persist data index=2 term=6 command=two 05:40:40,184 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader787: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 05:40:40,184 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Replicate message: identifier: MockIdentifier{value=two}, logIndex: 2, isSendImmediate: true 05:40:40,184 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 05:40:40,184 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 2 is present for follower follower788 05:40:40,184 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 648.4 ms 05:40:40,185 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Sending AppendEntries to follower follower788: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 05:40:40,185 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: false, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 05:40:40,185 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: awaiting more completions to resolve 1 synchronized callback(s) [WARN] [07/01/2025 17:40:40.185] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 05:40:40,185 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 05:40:40,185 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): Number of entries to be appended = 1 05:40:40,185 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 05:40:40,185 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): Append entry to log two 05:40:40,185 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower788: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 05:40:40,185 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): Log size is now 2 05:40:40,185 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower788, logLastIndex=2, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:40,185 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 648.6 ms 05:40:40,185 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 1 (of expected -1) 05:40:40,187 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,187 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,187 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 1.579 ms 05:40:40,187 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,187 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 2.287 ms 05:40:40,187 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,187 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader787: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=true} 05:40:40,187 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower788: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=false} 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@3b6ac117 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower788 updated: matchIndex: 2, nextIndex: 3 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader787 (Leader): Setting last applied to 2 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader787: Moving last applied index from 1 to 2 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader787: Applying state for log index 2 data two 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader787: applyState called: two 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader787: fakeSnapshot purging log to 1 for term 6 05:40:40,187 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 600.4 μs 05:40:40,187 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 1 05:40:40,188 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader787: Persist data index=3 term=6 command=three 05:40:40,188 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader787: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 05:40:40,188 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Replicate message: identifier: MockIdentifier{value=three}, logIndex: 3, isSendImmediate: true 05:40:40,188 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=1, applyTo=3] 05:40:40,188 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,188 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 3, leaderLastIndex: 3, leaderSnapShotIndex: 1 05:40:40,188 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 3 is present for follower follower788 05:40:40,188 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 241.1 μs 05:40:40,188 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Sending AppendEntries to follower follower788: AppendEntries{term=6, leaderId=leader787, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 05:40:40,188 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 2.680 μs 05:40:40,188 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: false, followerNextIndex: 2, leaderLastIndex: 3, leaderSnapShotIndex: 1 05:40:40,188 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,188 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: awaiting more completions to resolve 1 synchronized callback(s) 05:40:40,188 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): Commit index set to 2 05:40:40,188 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 2 (of expected -1) [WARN] [07/01/2025 17:40:40.188] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 05:40:40,188 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 2 05:40:40,188 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower788 (Follower): Setting last applied to 2 05:40:40,188 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower788: Moving last applied index from 1 to 2 05:40:40,188 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower788: Applying state for log index 2 data two 05:40:40,188 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower788: applyState called: two 05:40:40,188 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 1.501 ms 05:40:40,188 PM [follower788-writer-167] [DEBUG] EntryJournalV1 - follower788: updated meta to JournalMeta[replayFrom=1, applyTo=3] 05:40:40,188 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,188 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 160.5 μs 05:40:40,188 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:40,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - follower788: fakeSnapshot purging log to 1 for term 6 05:40:40,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader787, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 05:40:40,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): Number of entries to be appended = 1 05:40:40,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 05:40:40,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): Append entry to log three 05:40:40,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower788: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 05:40:40,189 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 299.4 μs 05:40:40,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): Log size is now 2 05:40:40,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower788, logLastIndex=3, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:40,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 2 (of expected -1) 05:40:40,190 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,190 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,190 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 1.975 ms 05:40:40,190 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,190 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader787: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=true} 05:40:40,190 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=2, lastAppliedTerm=6, lastIndex=3, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=1, mandatoryTrim=false] 05:40:40,190 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader787: captured snapshot at lastSequenceNumber: 4 05:40:40,190 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Persising snapshot at EntryInfo[index=2, term=6]/EntryInfo[index=3, term=6] 05:40:40,190 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader787: support() called 05:40:40,191 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Removed in-memory snapshotted entries, adjusted snaphsotIndex: 1 and term: 6 05:40:40,191 PM [leader787-%d553127] [DEBUG] RaftStorage - leader787: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-0B60DDE0.tmp 05:40:40,191 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@26c6be1f 05:40:40,191 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,191 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 2.033 ms 05:40:40,191 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:40,191 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 3 is present for follower follower788 05:40:40,191 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,191 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower788: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=false} 05:40:40,191 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower788 updated: matchIndex: 3, nextIndex: 4 05:40:40,191 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2 05:40:40,191 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader787 (Leader): Setting last applied to 3 05:40:40,191 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader787: Moving last applied index from 2 to 3 05:40:40,191 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader787: Applying state for log index 3 data three 05:40:40,191 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader787: applyState called: three 05:40:40,192 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 1.743 ms 05:40:40,192 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 4, leaderLastIndex: 3, leaderSnapShotIndex: 1 05:40:40,192 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): Commit index set to 3 05:40:40,192 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 3 (of expected -1) 05:40:40,192 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower788 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): 2, lastIndex(): 3 05:40:40,192 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower788 (Follower): Setting last applied to 3 05:40:40,192 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower788: Moving last applied index from 2 to 3 05:40:40,192 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower788: Applying state for log index 3 data three 05:40:40,192 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower788: applyState called: three 05:40:40,192 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=1, applyTo=4] 05:40:40,192 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,192 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 229.1 μs 05:40:40,192 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,192 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,192 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 1.021 ms 05:40:40,192 PM [follower788-writer-167] [DEBUG] EntryJournalV1 - follower788: updated meta to JournalMeta[replayFrom=1, applyTo=4] 05:40:40,192 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,192 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 76.30 μs 05:40:40,192 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:40,192 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader787: Persist data index=4 term=6 command=four 05:40:40,192 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader787: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 05:40:40,192 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,192 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Replicate message: identifier: MockIdentifier{value=four}, logIndex: 4, isSendImmediate: true 05:40:40,192 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 290.1 μs 05:40:40,192 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 4, leaderLastIndex: 4, leaderSnapShotIndex: 1 05:40:40,192 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 4 is present for follower follower788 05:40:40,192 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Sending AppendEntries to follower follower788: AppendEntries{term=6, leaderId=leader787, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 05:40:40,192 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: false, followerNextIndex: 2, leaderLastIndex: 4, leaderSnapShotIndex: 1 05:40:40,192 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader787, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 05:40:40,192 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: awaiting more completions to resolve 1 synchronized callback(s) 05:40:40,192 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): Number of entries to be appended = 1 05:40:40,192 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): After cleanup, lastIndex: 3, entries to be added from: 0 05:40:40,192 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): Append entry to log four 05:40:40,192 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower788: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} [WARN] [07/01/2025 17:40:40.192] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 05:40:40,193 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): Log size is now 3 05:40:40,193 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower788, logLastIndex=4, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:40,193 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 545.1 μs 05:40:40,193 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 3 (of expected -1) 05:40:40,193 PM [leader787-%d553127] [DEBUG] RaftStorage - leader787: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-0B60DDE0.v1 05:40:40,193 PM [leader787-%d553127] [DEBUG] RaftStorage - leader787: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-0B60DDE0.v1 05:40:40,193 PM [leader787-%d553127] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/TermInfo.properties 05:40:40,193 PM [leader787-%d553127] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/journal-v1-1.log 05:40:40,193 PM [leader787-%d553127] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/metadata-v1-1.log 05:40:40,193 PM [leader787-%d553127] [DEBUG] RaftStorage - leader787: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-07-01T17:40:40.190897632Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-0B60DDE0.v1, size=510, entryCompress=NONE, stateCompress=NONE} 05:40:40,193 PM [leader787-%d553127] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,193 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,193 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader787: snapshot is durable as of 2025-07-01T17:40:40.190897632Z 05:40:40,193 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader787: Snapshot success - sequence number: 4 05:40:40,193 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: awaiting more completions to resolve 1 synchronized callback(s) 05:40:40,194 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,194 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,194 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 1.421 ms 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,194 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:40,194 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 1.904 ms 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader787: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=true} 05:40:40,194 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 2.170 μs 05:40:40,194 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@5758d0a5 05:40:40,194 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower788: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=false} 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower788 updated: matchIndex: 4, nextIndex: 5 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Applying to log - commitIndex: 4, lastAppliedIndex: 3 05:40:40,194 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=4, applyTo=4] 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader787 (Leader): Setting last applied to 4 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader787: Moving last applied index from 3 to 4 05:40:40,194 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader787: Applying state for log index 4 data four 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader787: applyState called: four 05:40:40,194 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 96.86 μs 05:40:40,194 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 5, leaderLastIndex: 4, leaderSnapShotIndex: 1 05:40:40,194 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 77.60 μs 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,194 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=4, applyTo=5] 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorSnapshotMessageSupport - leader787: SnapshotComplete received 05:40:40,194 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,194 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 63.17 μs 05:40:40,194 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): Commit index set to 4 05:40:40,194 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,194 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 4 (of expected -1) 05:40:40,194 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 4, context.getLastApplied(): 3, lastIndex(): 4 05:40:40,194 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower788 (Follower): Setting last applied to 4 05:40:40,194 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower788: Moving last applied index from 3 to 4 05:40:40,194 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower788: Applying state for log index 4 data four 05:40:40,194 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower788: applyState called: four 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader787: Persist data index=5 term=6 command=five 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader787: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Replicate message: identifier: MockIdentifier{value=five}, logIndex: 5, isSendImmediate: true 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 5, leaderLastIndex: 5, leaderSnapShotIndex: 1 05:40:40,194 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 127.6 μs 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 5 is present for follower follower788 05:40:40,194 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Sending AppendEntries to follower follower788: AppendEntries{term=6, leaderId=leader787, prevLogIndex=4, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 05:40:40,194 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 470.4 μs 05:40:40,195 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: false, followerNextIndex: 2, leaderLastIndex: 5, leaderSnapShotIndex: 1 05:40:40,195 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: awaiting more completions to resolve 1 synchronized callback(s) 05:40:40,195 PM [follower788-writer-167] [DEBUG] EntryJournalV1 - follower788: updated meta to JournalMeta[replayFrom=1, applyTo=5] 05:40:40,195 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,195 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 101.9 μs 05:40:40,195 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:40,195 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,195 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader787, prevLogIndex=4, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 05:40:40,195 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): Number of entries to be appended = 1 [WARN] [07/01/2025 17:40:40.195] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 05:40:40,195 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): After cleanup, lastIndex: 4, entries to be added from: 0 05:40:40,195 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): Append entry to log five 05:40:40,195 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower788: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 05:40:40,195 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): Log size is now 4 05:40:40,195 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 249.9 μs 05:40:40,195 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower788 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower788, logLastIndex=5, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:40,195 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 4 (of expected -1) 05:40:40,195 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,195 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,195 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 909.1 μs 05:40:40,195 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,195 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader787: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=true} 05:40:40,195 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@5cdc4bde 05:40:40,195 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower788 updated: matchIndex: 5, nextIndex: 6 05:40:40,195 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Applying to log - commitIndex: 5, lastAppliedIndex: 4 05:40:40,195 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader787 (Leader): Setting last applied to 5 05:40:40,195 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader787: Moving last applied index from 4 to 5 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader787: Applying state for log index 5 data five 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader787: applyState called: five 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 6, leaderLastIndex: 5, leaderSnapShotIndex: 1 05:40:40,196 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,196 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 212.3 μs 05:40:40,196 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 716.5 μs 05:40:40,196 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader787: Persist data index=6 term=6 command=six 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader787: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=true} 05:40:40,196 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=4, applyTo=6] 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,196 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Replicate message: identifier: MockIdentifier{value=six}, logIndex: 6, isSendImmediate: true 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower788: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=false} 05:40:40,196 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 85.42 μs 05:40:40,196 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 1.480 μs 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 6, leaderLastIndex: 6, leaderSnapShotIndex: 1 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 6 is present for follower follower788 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Sending AppendEntries to follower follower788: AppendEntries{term=6, leaderId=leader787, prevLogIndex=5, prevLogTerm=6, leaderCommit=5, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower788 (Follower): Commit index set to 5 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: false, followerNextIndex: 2, leaderLastIndex: 6, leaderSnapShotIndex: 1 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 5 (of expected -1) 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower788 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 5, context.getLastApplied(): 4, lastIndex(): 5 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower788 (Follower): Setting last applied to 5 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower788: Moving last applied index from 4 to 5 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower788: Applying state for log index 5 data five 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower788: applyState called: five 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,196 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: awaiting more completions to resolve 1 synchronized callback(s) [WARN] [07/01/2025 17:40:40.196] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 05:40:40,196 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 341.5 μs 05:40:40,196 PM [follower788-writer-167] [DEBUG] EntryJournalV1 - follower788: updated meta to JournalMeta[replayFrom=1, applyTo=6] 05:40:40,196 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,196 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 74.96 μs 05:40:40,196 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower788 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader787, prevLogIndex=5, prevLogTerm=6, leaderCommit=5, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower788 (Follower): Number of entries to be appended = 1 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower788 (Follower): After cleanup, lastIndex: 5, entries to be added from: 0 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower788 (Follower): Append entry to log six 05:40:40,196 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower788: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=true} 05:40:40,196 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 172.6 μs 05:40:40,197 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower788 (Follower): Log size is now 5 05:40:40,197 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower788 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower788, logLastIndex=6, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:40,197 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 5 (of expected -1) 05:40:40,197 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,197 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,197 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 1.708 ms 05:40:40,197 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,197 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader787: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=true} 05:40:40,197 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader787: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@62b69729 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 6 is present for follower follower788 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower788 updated: matchIndex: 6, nextIndex: 7 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Applying to log - commitIndex: 6, lastAppliedIndex: 5 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader787 (Leader): Setting last applied to 6 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader787: Moving last applied index from 5 to 6 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader787: Applying state for log index 6 data six 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader787: applyState called: six 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 7, leaderLastIndex: 6, leaderSnapShotIndex: 1 05:40:40,198 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 323.9 μs 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower788 (Follower): Commit index set to 6 05:40:40,198 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=4, applyTo=7] 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 6 (of expected -1) 05:40:40,198 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower788 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 6, context.getLastApplied(): 5, lastIndex(): 6 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower788 (Follower): Setting last applied to 6 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower788: Moving last applied index from 5 to 6 05:40:40,198 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 76.64 μs 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower788: Applying state for log index 6 data six 05:40:40,198 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,198 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower788: applyState called: six 05:40:40,198 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,199 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,199 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 3.143 ms 05:40:40,199 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,199 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 2.380 μs 05:40:40,199 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower788: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=false} 05:40:40,200 PM [follower788-writer-167] [DEBUG] EntryJournalV1 - follower788: updated meta to JournalMeta[replayFrom=1, applyTo=7] 05:40:40,200 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,200 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,200 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 137.1 μs 05:40:40,200 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands [WARN] [07/01/2025 17:40:40.213] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 05:40:40,285 PM [awaitility-thread] [DEBUG] RaftStorage - leader787: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-0B60DDE0.v1 05:40:40,285 PM [awaitility-thread] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/TermInfo.properties 05:40:40,285 PM [awaitility-thread] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/journal-v1-1.log 05:40:40,285 PM [awaitility-thread] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/metadata-v1-1.log 05:40:40,285 PM [awaitility-thread] [DEBUG] RaftStorage - leader787: picked SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-07-01T17:40:40.190897632Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-0B60DDE0.v1, size=510, entryCompress=NONE, stateCompress=NONE} as the latest file 05:40:40,286 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderInstallsSnapshotWithRestartedFollowerDuringSnapshotInstallation: sending 1 more payload to trigger second snapshot 05:40:40,286 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader787: Persist data index=7 term=6 command=seven 05:40:40,286 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader787: Append log entry and persist JournaledLogEntry{index=7, term=6, command=seven, pending=true} 05:40:40,286 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 88.31 ms 05:40:40,287 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Replicate message: identifier: MockIdentifier{value=seven}, logIndex: 7, isSendImmediate: true 05:40:40,287 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 7, leaderLastIndex: 7, leaderSnapShotIndex: 1 05:40:40,287 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): sendAppendEntries: 7 is present for follower follower788 05:40:40,287 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Sending AppendEntries to follower follower788: AppendEntries{term=6, leaderId=leader787, prevLogIndex=6, prevLogTerm=6, leaderCommit=6, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=7, term=6, command=seven, pending=true}]} 05:40:40,287 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader787, prevLogIndex=6, prevLogTerm=6, leaderCommit=6, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=7, term=6, command=seven, pending=true}]} 05:40:40,287 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): Number of entries to be appended = 1 05:40:40,287 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): After cleanup, lastIndex: 6, entries to be added from: 0 05:40:40,287 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): Append entry to log seven 05:40:40,287 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower788: Append log entry and persist JournaledLogEntry{index=7, term=6, command=seven, pending=true} 05:40:40,287 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): Log size is now 6 05:40:40,287 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower788 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower788, logLastIndex=7, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:40,287 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 6 (of expected -1) 05:40:40,287 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 87.48 ms 05:40:40,287 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: false, followerNextIndex: 2, leaderLastIndex: 7, leaderSnapShotIndex: 1 05:40:40,287 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: awaiting more completions to resolve 1 synchronized callback(s) [WARN] [07/01/2025 17:40:40.287] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 05:40:40,289 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,289 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 2.418 ms 05:40:40,289 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,289 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,289 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader787: persist complete JournaledLogEntry{index=7, term=6, command=seven, pending=true} 05:40:40,289 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=6, lastAppliedTerm=6, lastIndex=7, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=1, mandatoryTrim=false] 05:40:40,289 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader787: captured snapshot at lastSequenceNumber: 8 05:40:40,289 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Persising snapshot at EntryInfo[index=6, term=6]/EntryInfo[index=7, term=6] 05:40:40,289 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader787: support() called 05:40:40,290 PM [leader787-%d553128] [DEBUG] RaftStorage - leader787: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-11475A60.tmp 05:40:40,290 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,290 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 2.900 ms 05:40:40,290 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:40,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader787: log size 6 exceeds the snapshot batch count 4 - doing snapshotPreCommit with index 6 05:40:40,290 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Removed in-memory snapshotted entries, adjusted snaphsotIndex: 6 and term: 6 05:40:40,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@6bfc6c55 05:40:40,290 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,290 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower788: persist complete JournaledLogEntry{index=7, term=6, command=seven, pending=false} 05:40:40,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower788 updated: matchIndex: 7, nextIndex: 8 05:40:40,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Applying to log - commitIndex: 7, lastAppliedIndex: 6 05:40:40,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader787 (Leader): Setting last applied to 7 05:40:40,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader787: Moving last applied index from 6 to 7 05:40:40,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader787: Applying state for log index 7 data seven 05:40:40,290 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader787: applyState called: seven 05:40:40,291 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower788: active: true, followerNextIndex: 8, leaderLastIndex: 7, leaderSnapShotIndex: 6 05:40:40,291 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 1.834 ms 05:40:40,291 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower788 (Follower): Commit index set to 7 05:40:40,291 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 7 (of expected -1) 05:40:40,291 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower788 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 7, context.getLastApplied(): 6, lastIndex(): 7 05:40:40,291 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower788 (Follower): Setting last applied to 7 05:40:40,291 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower788: Moving last applied index from 6 to 7 05:40:40,291 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower788: Applying state for log index 7 data seven 05:40:40,291 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower788: applyState called: seven 05:40:40,291 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 850.6 μs 05:40:40,291 PM [follower788-writer-167] [DEBUG] EntryJournalV1 - follower788: updated meta to JournalMeta[replayFrom=1, applyTo=8] 05:40:40,291 PM [follower788-writer-167] [DEBUG] RaftStorageCompleter - follower788: 1 completion(s) pending 05:40:40,291 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 168.4 μs 05:40:40,291 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower788: running 1 completion(s) 05:40:40,291 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 05:40:40,291 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=4, applyTo=8] 05:40:40,291 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,291 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 726.6 μs 05:40:40,291 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,291 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,292 PM [leader787-%d553128] [DEBUG] RaftStorage - leader787: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-11475A60.v1 05:40:40,292 PM [leader787-%d553128] [DEBUG] RaftStorage - leader787: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-0B60DDE0.v1 05:40:40,292 PM [leader787-%d553128] [DEBUG] RaftStorage - leader787: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-11475A60.v1 05:40:40,292 PM [leader787-%d553128] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/TermInfo.properties 05:40:40,292 PM [leader787-%d553128] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/journal-v1-1.log 05:40:40,292 PM [leader787-%d553128] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/metadata-v1-1.log 05:40:40,292 PM [leader787-%d553128] [DEBUG] RaftStorage - leader787: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-07-01T17:40:40.190897632Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-0B60DDE0.v1, size=510, entryCompress=NONE, stateCompress=NONE} 05:40:40,292 PM [leader787-%d553128] [DEBUG] RaftStorage - leader787: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-07-01T17:40:40.289888864Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-11475A60.v1, size=578, entryCompress=NONE, stateCompress=NONE} 05:40:40,292 PM [leader787-%d553128] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,292 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,292 PM [test-pekko.actor.default-dispatcher-15] [INFO] SnapshotManager - leader787: snapshot is durable as of 2025-07-01T17:40:40.289888864Z 05:40:40,293 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - leader787: Snapshot success - sequence number: 8 05:40:40,293 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorSnapshotMessageSupport - leader787: SnapshotComplete received 05:40:40,293 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 1.243 ms 05:40:40,293 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=8, applyTo=8] 05:40:40,293 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,293 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 111.4 μs 05:40:40,293 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,293 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) [WARN] [07/01/2025 17:40:40.332] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [07/01/2025 17:40:40.452] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower789] received dead letter from TestActor[pekko://test/user/leader787]: AppendEntries{term=6, leaderId=leader787, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 05:40:40,452 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 7 (of expected -1) 05:40:40,488 PM [awaitility-thread] [DEBUG] RaftStorage - leader787: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-11475A60.v1 05:40:40,488 PM [awaitility-thread] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/TermInfo.properties 05:40:40,488 PM [awaitility-thread] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/journal-v1-1.log 05:40:40,488 PM [awaitility-thread] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/metadata-v1-1.log 05:40:40,488 PM [awaitility-thread] [DEBUG] RaftStorage - leader787: picked SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-07-01T17:40:40.289888864Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-11475A60.v1, size=578, entryCompress=NONE, stateCompress=NONE} as the latest file [WARN] [07/01/2025 17:40:40.491] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower789]: org.opendaylight.controller.cluster.common.actor.Monitor@40e4a2cd 05:40:40,492 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower789: Starting recovery with journal batch size 1000 05:40:40,492 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower789: started executor 05:40:40,492 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower789: started executor 05:40:40,493 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower789: metadata journal open: replayFrom=1 applyTo=2 05:40:40,493 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower789: journal open: firstIndex=1 lastIndex=2 replayFrom=1 fileEntries=0 05:40:40,493 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower789: journal open: applyTo=2 05:40:40,494 PM [follower789-writer-170] [DEBUG] JournalWriteTask - follower789: journal writer started 05:40:40,494 PM [follower789-writer-170] [DEBUG] JournalWriteTask - follower789: waiting to receive commands 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower789: Set currentTerm=6, votedFor=leader787 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/TermInfo.properties 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/journal-v1-1.log 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/metadata-v1-1.log 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower789: no eligible files found 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower789: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower789: restoring local TermInfo{term=6, votedFor=leader787} 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower789: Set currentTerm=6, votedFor=leader787 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower789: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@ebdb9fa 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower789: Moving last applied index from -1 to -1 05:40:40,495 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower789: applying entries up to 2 05:40:40,496 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower789: recovered journal DefaultLogEntry[index=0, term=6, command=zero] 05:40:40,497 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower789: recovered journal DefaultLogEntry[index=1, term=6, command=one] 05:40:40,497 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower789: Moving last applied index from -1 to 1 05:40:40,497 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower789: journal recovery completedin 1.817 ms with journalIndex=1 05:40:40,572 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 7 (of expected -1) 05:40:40,573 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower789: Last sync leader does not match current leader leader787, need to catch up to -1 05:40:40,573 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower789: fakeSnapshot purging log to 0 for term 6 05:40:40,573 PM [test-pekko.actor.default-dispatcher-11] [WARN] AbstractLeader - leader787 (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply{term=6, success=true, followerId=follower789, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, timeSinceLastActivity : 840, lastApplied : 7, commitIndex : 7 05:40:40,573 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractLeader - leader787 (Leader): Initiating install snapshot to follower follower789: follower nextIndex: 2, leader snapshotIndex: 6, leader lastIndex: 7, leader log size: 1 05:40:40,573 PM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - leader787: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=7, lastAppliedTerm=6, lastIndex=7, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower789 05:40:40,573 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader787: lastSequenceNumber prior to capture: 9 [WARN] [07/01/2025 17:40:40.573] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower789] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower789, initialSyncDone=false] 05:40:40,573 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader787: support() called 05:40:40,574 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower789: Lagging -2 entries behind leader leader787 and reached 1 (of expected -1) [WARN] [07/01/2025 17:40:40.574] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower789] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower789, initialSyncDone=true] 05:40:40,574 PM [leader787-%d553129] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,575 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,576 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Persising snapshot at EntryInfo[index=7, term=6]/EntryInfo[index=7, term=6] 05:40:40,576 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader787: support() called 05:40:40,576 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Removed in-memory snapshotted entries, adjusted snaphsotIndex: 6 and term: 6 05:40:40,576 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): sendInstallSnapshot 05:40:40,576 PM [leader787-%d553130] [DEBUG] RaftStorage - leader787: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-225D7AD1.tmp 05:40:40,576 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] LeaderInstallSnapshotState - leader787 (Leader): Snapshot 396 bytes, total chunks to send: 1 05:40:40,576 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] LeaderInstallSnapshotState - leader787 (Leader): Next chunk: total length=396, offset=0, size=396, hashCode=-153134944 05:40:40,576 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): next snapshot chunk size for follower follower789: 396 05:40:40,576 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): InstallSnapshot sent to follower Vector(user, follower789), Chunk: 1/1 05:40:40,577 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower789 (Follower): handleInstallSnapshot: InstallSnapshot{term=6, leaderId=leader787, lastIncludedIndex=7, lastIncludedTerm=6, datasize=396, chunk=1/1, lastChunkHashCode=OptionalInt[-1]} 05:40:40,577 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower789: Lagging 6 entries behind leader leader787 and reached 1 (of expected -1) 05:40:40,577 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotTracker - follower789 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 05:40:40,577 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower789 (Follower): Snapshot received from leader: leader787 05:40:40,577 PM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - follower789: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=396}} 05:40:40,577 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower789: support() called 05:40:40,578 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower789: Converted InstallSnapshot from leader: leader787 to state 05:40:40,578 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower789: lastSequenceNumber prior to persisting applied snapshot: 3 05:40:40,578 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower789: support() called 05:40:40,578 PM [follower789-%d438645] [DEBUG] RaftStorage - follower789: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/snapshot-0000000068641D98-227A5EDD.tmp 05:40:40,582 PM [leader787-%d553130] [DEBUG] RaftStorage - leader787: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-225D7AD1.v1 05:40:40,582 PM [leader787-%d553130] [DEBUG] RaftStorage - leader787: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-225D7AD1.v1 05:40:40,582 PM [leader787-%d553130] [DEBUG] RaftStorage - leader787: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-11475A60.v1 05:40:40,582 PM [leader787-%d553130] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/TermInfo.properties 05:40:40,582 PM [leader787-%d553130] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/journal-v1-1.log 05:40:40,582 PM [leader787-%d553130] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/metadata-v1-1.log 05:40:40,582 PM [leader787-%d553130] [DEBUG] RaftStorage - leader787: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-07-01T17:40:40.289888864Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-11475A60.v1, size=578, entryCompress=NONE, stateCompress=NONE} 05:40:40,582 PM [leader787-%d553130] [DEBUG] RaftStorage - leader787: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=7, term=6], timestamp=2025-07-01T17:40:40.576551633Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-225D7AD1.v1, size=460, entryCompress=NONE, stateCompress=NONE} 05:40:40,582 PM [leader787-%d553130] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,582 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,582 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: snapshot is durable as of 2025-07-01T17:40:40.576551633Z 05:40:40,582 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader787: Snapshot success - sequence number: 9 05:40:40,582 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorSnapshotMessageSupport - leader787: SnapshotComplete received 05:40:40,582 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 289.5 ms 05:40:40,582 PM [follower789-%d438645] [DEBUG] RaftStorage - follower789: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/snapshot-0000000068641D98-227A5EDD.v1 05:40:40,583 PM [follower789-%d438645] [DEBUG] RaftStorage - follower789: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/snapshot-0000000068641D98-227A5EDD.v1 05:40:40,583 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=9, applyTo=8] 05:40:40,583 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,583 PM [follower789-%d438645] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/TermInfo.properties 05:40:40,583 PM [follower789-%d438645] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/journal-v1-1.log 05:40:40,583 PM [follower789-%d438645] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/metadata-v1-1.log 05:40:40,583 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 257.7 μs 05:40:40,583 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,583 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,583 PM [follower789-%d438645] [DEBUG] RaftStorage - follower789: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=7, term=6], timestamp=2025-07-01T17:40:40.578445021Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/snapshot-0000000068641D98-227A5EDD.v1, size=460, entryCompress=NONE, stateCompress=NONE} 05:40:40,583 PM [follower789-%d438645] [DEBUG] RaftStorageCompleter - follower789: 1 completion(s) pending 05:40:40,583 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower789: running 1 completion(s) 05:40:40,583 PM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - follower789: snapshot is durable as of 2025-07-01T17:40:40.578445021Z 05:40:40,583 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower789: Snapshot success - sequence number: 3 05:40:40,583 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower789: support() called 05:40:40,583 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower789: applySnapshot called 05:40:40,583 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower789 (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=6, followerId=follower789, chunkIndex=1, success=true} 05:40:40,583 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorSnapshotMessageSupport - follower789: SnapshotComplete received 05:40:40,583 PM [follower789-writer-170] [DEBUG] JournalWriteTask - follower789: received 1 commands after 89.67 ms 05:40:40,583 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=6, followerId=follower789, chunkIndex=1, success=true} 05:40:40,583 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractLeader - leader787 (Leader): Snapshot successfully installed on follower follower789 (last chunk 1) - matchIndex set to 7, nextIndex set to 8 05:40:40,583 PM [follower789-writer-170] [DEBUG] EntryJournalV1 - follower789: updated meta to JournalMeta[replayFrom=3, applyTo=2] 05:40:40,583 PM [follower789-writer-170] [DEBUG] RaftStorageCompleter - follower789: 1 completion(s) pending 05:40:40,583 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower789: running 1 completion(s) 05:40:40,583 PM [follower789-writer-170] [DEBUG] JournalWriteTask - follower789: completed 1 commands in 197.7 μs 05:40:40,583 PM [follower789-writer-170] [DEBUG] JournalWriteTask - follower789: waiting to receive commands 05:40:40,594 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower789: terminating thread VirtualThread[#951,follower789-writer-170]/waiting 05:40:40,595 PM [follower789-writer-170] [DEBUG] JournalWriteTask - follower789: received 1 commands after 11.11 ms 05:40:40,595 PM [follower789-writer-170] [DEBUG] JournalWriteTask - follower789: completed 1 commands in 13.47 μs 05:40:40,595 PM [follower789-writer-170] [DEBUG] JournalWriteTask - follower789: journal writer stopped 05:40:40,595 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower789: journal closed 05:40:40,595 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower789: journal closed 05:40:40,595 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: stopped executor with 0 remaining tasks 05:40:40,595 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: stopped executor with 0 remaining tasks [WARN] [07/01/2025 17:40:40.599] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower789]: org.opendaylight.controller.cluster.common.actor.Monitor@13b07595 05:40:40,600 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower789: Starting recovery with journal batch size 1000 05:40:40,600 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower789: started executor 05:40:40,600 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower789: started executor 05:40:40,601 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower789: metadata journal open: replayFrom=3 applyTo=2 05:40:40,601 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower789: journal open: firstIndex=1 lastIndex=2 replayFrom=3 fileEntries=0 05:40:40,601 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower789: journal open: applyTo=2 05:40:40,601 PM [follower789-writer-171] [DEBUG] JournalWriteTask - follower789: journal writer started 05:40:40,601 PM [follower789-writer-171] [DEBUG] JournalWriteTask - follower789: waiting to receive commands 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower789: Set currentTerm=6, votedFor=leader787 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/TermInfo.properties 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/journal-v1-1.log 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/metadata-v1-1.log 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: no eligible files found 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower789: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower789: restoring local TermInfo{term=6, votedFor=leader787} 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower789: Set currentTerm=6, votedFor=leader787 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower789: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@292afb5a 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower789: Moving last applied index from -1 to -1 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower789: applying entries up to 2 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower789: Moving last applied index from -1 to -1 05:40:40,603 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower789: journal recovery completedin 338.0 μs with journalIndex=3 05:40:40,693 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower789 (Follower): The followers log is empty and the senders prevLogIndex is 7 05:40:40,693 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower788: Lagging 0 entries behind leader leader787 and reached 7 (of expected -1) 05:40:40,693 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - follower788: fakeSnapshot purging log to 6 for term 6 05:40:40,693 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower789 (Follower): Follower is out-of-sync so sending negative reply: AppendEntriesReply{term=6, success=false, followerId=follower789, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:40,693 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower789: Last sync leader does not match current leader leader787, need to catch up to 7 [WARN] [07/01/2025 17:40:40.693] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower789] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower789, initialSyncDone=false] 05:40:40,693 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - received unsuccessful reply: AppendEntriesReply{term=6, success=false, followerId=follower789, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, leader snapshotIndex: 6, snapshotTerm: 6, replicatedToAllIndex: 6 05:40:40,693 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower789 updated: matchIndex: -1, nextIndex: 0 05:40:40,693 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractLeader - leader787 (Leader): follower follower789 appears to be behind the leader from the last snapshot - updated: matchIndex: -1, nextIndex: 0 05:40:40,693 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): Checking sendAppendEntries for follower follower789: active: true, followerNextIndex: 0, leaderLastIndex: 7, leaderSnapShotIndex: 6 05:40:40,693 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractLeader - leader787 (Leader): Initiating install snapshot to follower follower789: follower nextIndex: 0, leader snapshotIndex: 6, leader lastIndex: 7, leader log size: 1 05:40:40,693 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=7, lastAppliedTerm=6, lastIndex=7, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower789 05:40:40,693 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader787: lastSequenceNumber prior to capture: 9 05:40:40,693 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader787: support() called 05:40:40,694 PM [leader787-%d553131] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,694 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower789 (Follower): Cannot append entries because the replicatedToAllIndex 6 does not appear to be in the in-memory journal - lastIndex: -1, snapshotIndex: -1, snapshotTerm: -1 05:40:40,694 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower789 (Follower): Follower is out-of-sync so sending negative reply: AppendEntriesReply{term=6, success=false, followerId=follower789, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:40,695 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,695 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Persising snapshot at EntryInfo[index=7, term=6]/EntryInfo[index=7, term=6] 05:40:40,695 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader787: support() called 05:40:40,695 PM [leader787-%d553132] [DEBUG] RaftStorage - leader787: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-29726EFC.tmp 05:40:40,696 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: Removed in-memory snapshotted entries, adjusted snaphsotIndex: 6 and term: 6 05:40:40,696 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): sendInstallSnapshot 05:40:40,696 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] LeaderInstallSnapshotState - leader787 (Leader): Snapshot 396 bytes, total chunks to send: 1 05:40:40,696 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] LeaderInstallSnapshotState - leader787 (Leader): Next chunk: total length=396, offset=0, size=396, hashCode=-153134944 05:40:40,696 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): next snapshot chunk size for follower follower789: 396 05:40:40,696 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): InstallSnapshot sent to follower Vector(user, follower789), Chunk: 1/1 05:40:40,696 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower789 (Follower): handleInstallSnapshot: InstallSnapshot{term=6, leaderId=leader787, lastIncludedIndex=7, lastIncludedTerm=6, datasize=396, chunk=1/1, lastChunkHashCode=OptionalInt[-1]} 05:40:40,696 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotTracker - follower789 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 05:40:40,696 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractLeader - leader787 (Leader): handleAppendEntriesReply - received unsuccessful reply: AppendEntriesReply{term=6, success=false, followerId=follower789, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, leader snapshotIndex: 6, snapshotTerm: 6, replicatedToAllIndex: 6 05:40:40,696 PM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - follower789 (Follower): Snapshot received from leader: leader787 05:40:40,696 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractLeader - leader787 (Leader): follower follower789 appears to be behind the leader from the last snapshot - updated: matchIndex: -1, nextIndex: 0 05:40:40,696 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - follower789: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=396}} 05:40:40,696 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower789: support() called 05:40:40,697 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower789: Converted InstallSnapshot from leader: leader787 to state 05:40:40,697 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower789: lastSequenceNumber prior to persisting applied snapshot: 3 05:40:40,697 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower789: support() called 05:40:40,697 PM [leader787-%d553132] [DEBUG] RaftStorage - leader787: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-29726EFC.v1 05:40:40,698 PM [follower789-%d180976] [DEBUG] RaftStorage - follower789: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/snapshot-0000000068641D98-299AE4DD.tmp 05:40:40,698 PM [leader787-%d553132] [DEBUG] RaftStorage - leader787: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-225D7AD1.v1 05:40:40,698 PM [leader787-%d553132] [DEBUG] RaftStorage - leader787: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-29726EFC.v1 05:40:40,698 PM [leader787-%d553132] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/TermInfo.properties 05:40:40,698 PM [leader787-%d553132] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/journal-v1-1.log 05:40:40,698 PM [leader787-%d553132] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/metadata-v1-1.log 05:40:40,698 PM [leader787-%d553132] [DEBUG] RaftStorage - leader787: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=7, term=6], timestamp=2025-07-01T17:40:40.576551633Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-225D7AD1.v1, size=460, entryCompress=NONE, stateCompress=NONE} 05:40:40,698 PM [leader787-%d553132] [DEBUG] RaftStorage - leader787: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=7, term=6], timestamp=2025-07-01T17:40:40.695365372Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/leader787/snapshot-0000000068641D98-29726EFC.v1, size=460, entryCompress=NONE, stateCompress=NONE} 05:40:40,698 PM [leader787-%d553132] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,699 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,699 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader787: snapshot is durable as of 2025-07-01T17:40:40.695365372Z 05:40:40,699 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader787: Snapshot success - sequence number: 9 05:40:40,699 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 116.0 ms 05:40:40,699 PM [leader787-writer-169] [DEBUG] EntryJournalV1 - leader787: first entry 9 already includes 9 05:40:40,699 PM [leader787-writer-169] [DEBUG] RaftStorageCompleter - leader787: 1 completion(s) pending 05:40:40,699 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 46.61 μs 05:40:40,699 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 05:40:40,699 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader787: running 1 completion(s) 05:40:40,699 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorSnapshotMessageSupport - leader787: SnapshotComplete received 05:40:40,700 PM [follower789-%d180976] [DEBUG] RaftStorage - follower789: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/snapshot-0000000068641D98-299AE4DD.v1 05:40:40,700 PM [follower789-%d180976] [DEBUG] RaftStorage - follower789: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/snapshot-0000000068641D98-299AE4DD.v1 05:40:40,700 PM [follower789-%d180976] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/TermInfo.properties 05:40:40,700 PM [follower789-%d180976] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/journal-v1-1.log 05:40:40,700 PM [follower789-%d180976] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/metadata-v1-1.log 05:40:40,700 PM [follower789-%d180976] [DEBUG] RaftStorage - follower789: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=7, term=6], timestamp=2025-07-01T17:40:40.698016989Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/snapshot-0000000068641D98-299AE4DD.v1, size=460, entryCompress=NONE, stateCompress=NONE} 05:40:40,700 PM [follower789-%d180976] [DEBUG] RaftStorageCompleter - follower789: 1 completion(s) pending 05:40:40,700 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower789: running 1 completion(s) 05:40:40,700 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - follower789: snapshot is durable as of 2025-07-01T17:40:40.698016989Z 05:40:40,700 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower789: Snapshot success - sequence number: 3 05:40:40,700 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower789: support() called 05:40:40,701 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower789: applySnapshot called 05:40:40,701 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower789 (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=6, followerId=follower789, chunkIndex=1, success=true} 05:40:40,701 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorSnapshotMessageSupport - follower789: SnapshotComplete received 05:40:40,701 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader787 (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=6, followerId=follower789, chunkIndex=1, success=true} 05:40:40,701 PM [follower789-writer-171] [DEBUG] JournalWriteTask - follower789: received 1 commands after 99.27 ms 05:40:40,701 PM [follower789-writer-171] [DEBUG] EntryJournalV1 - follower789: first entry 3 already includes 3 05:40:40,701 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractLeader - leader787 (Leader): Snapshot successfully installed on follower follower789 (last chunk 1) - matchIndex set to 7, nextIndex set to 8 05:40:40,701 PM [follower789-writer-171] [DEBUG] RaftStorageCompleter - follower789: 1 completion(s) pending 05:40:40,701 PM [follower789-writer-171] [DEBUG] JournalWriteTask - follower789: completed 1 commands in 67.93 μs 05:40:40,701 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower789: running 1 completion(s) 05:40:40,701 PM [follower789-writer-171] [DEBUG] JournalWriteTask - follower789: waiting to receive commands 05:40:40,702 PM [awaitility-thread] [DEBUG] RaftStorage - follower789: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/snapshot-0000000068641D98-299AE4DD.v1 05:40:40,702 PM [awaitility-thread] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/TermInfo.properties 05:40:40,703 PM [awaitility-thread] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/journal-v1-1.log 05:40:40,703 PM [awaitility-thread] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/metadata-v1-1.log 05:40:40,703 PM [awaitility-thread] [DEBUG] RaftStorage - follower789: picked SnapshotFileV1{lastIncluded=EntryInfo[index=7, term=6], timestamp=2025-07-01T17:40:40.698016989Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest3396309691704172328/follower789/snapshot-0000000068641D98-299AE4DD.v1, size=460, entryCompress=NONE, stateCompress=NONE} as the latest file 05:40:40,703 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower788-collector790#892064606] 05:40:40,714 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower788] 05:40:40,714 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower788: terminating thread VirtualThread[#943,follower788-writer-167]/waiting 05:40:40,714 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: received 1 commands after 423.0 ms 05:40:40,714 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 12.79 μs 05:40:40,714 PM [follower788-writer-167] [DEBUG] JournalWriteTask - follower788: journal writer stopped 05:40:40,714 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower788: journal closed 05:40:40,714 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower788: journal closed 05:40:40,714 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower788: stopped executor with 0 remaining tasks 05:40:40,714 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower788: stopped executor with 0 remaining tasks 05:40:40,715 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower789-collector791#1387910415] 05:40:40,715 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower789] 05:40:40,715 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader787-collector792#1343503017] 05:40:40,715 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader787] 05:40:40,715 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - leader787: terminating thread VirtualThread[#945,leader787-writer-169]/waiting 05:40:40,715 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: received 1 commands after 16.66 ms 05:40:40,716 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 11.29 μs 05:40:40,716 PM [leader787-writer-169] [DEBUG] JournalWriteTask - leader787: journal writer stopped 05:40:40,716 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader787: journal closed 05:40:40,716 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - leader787: journal closed 05:40:40,716 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader787: stopped executor with 0 remaining tasks 05:40:40,716 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader787: stopped executor with 0 remaining tasks 05:40:40,716 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower789-collector793#2134512374] 05:40:40,716 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower789] 05:40:40,716 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower789-collector794#-1752249161] 05:40:40,716 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower789] 05:40:40,716 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower789: terminating thread VirtualThread[#955,follower789-writer-171]/waiting 05:40:40,716 PM [follower789-writer-171] [DEBUG] JournalWriteTask - follower789: received 1 commands after 15.60 ms 05:40:40,717 PM [follower789-writer-171] [DEBUG] JournalWriteTask - follower789: completed 1 commands in 8.950 μs 05:40:40,717 PM [follower789-writer-171] [DEBUG] JournalWriteTask - follower789: journal writer stopped 05:40:40,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower789: journal closed 05:40:40,717 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower789: journal closed 05:40:40,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower789: stopped executor with 0 remaining tasks 05:40:40,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower789: stopped executor with 0 remaining tasks 05:40:40,719 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotTriggeredByMemoryThresholdExceededWithLaggingFollower starting [WARN] [07/01/2025 17:40:40.720] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower799]: org.opendaylight.controller.cluster.common.actor.Monitor@22ea496f 05:40:40,721 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower799: Starting recovery with journal batch size 1000 05:40:40,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower799: started executor 05:40:40,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower799: started executor 05:40:40,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower799: metadata journal open: replayFrom=1 applyTo=0 05:40:40,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower799: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:40,722 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower799: journal open: applyTo=0 05:40:40,722 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: journal writer started 05:40:40,722 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: waiting to receive commands [WARN] [07/01/2025 17:40:40.723] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower800]: org.opendaylight.controller.cluster.common.actor.Monitor@37b6c845 05:40:40,723 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest12029400409825354904/follower799/TermInfo.properties does not exist 05:40:40,723 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower799: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest12029400409825354904/follower799/journal-v1-1.log 05:40:40,723 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower799: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest12029400409825354904/follower799/metadata-v1-1.log 05:40:40,723 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower799: no eligible files found 05:40:40,723 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower799: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:40,723 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower800: Starting recovery with journal batch size 1000 05:40:40,723 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower800: started executor 05:40:40,723 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower800: started executor 05:40:40,723 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower800: metadata journal open: replayFrom=1 applyTo=0 05:40:40,724 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower800: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:40,724 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower800: journal open: applyTo=0 05:40:40,724 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: journal writer started 05:40:40,724 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: waiting to receive commands 05:40:40,725 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest12029400409825354904/follower800/TermInfo.properties does not exist 05:40:40,725 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower800: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest12029400409825354904/follower800/journal-v1-1.log 05:40:40,725 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower800: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest12029400409825354904/follower800/metadata-v1-1.log 05:40:40,725 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower800: no eligible files found 05:40:40,725 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower800: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/01/2025 17:40:40.725] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader798]: org.opendaylight.controller.cluster.common.actor.Monitor@2875a6c8 05:40:40,725 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower799: Set currentTerm=0, votedFor=null 05:40:40,725 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower799: Local TermInfo store seeded with TermInfo{term=0} 05:40:40,725 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower799: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@3eeb2fa3 05:40:40,725 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower799: Moving last applied index from -1 to -1 05:40:40,726 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower799: empty journal: appending 0 entries 05:40:40,726 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower799: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:40,726 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower799: journal recovery completedin 308.5 μs with journalIndex=1 05:40:40,726 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader798: Starting recovery with journal batch size 1000 05:40:40,727 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader798: started executor 05:40:40,727 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader798: started executor 05:40:40,727 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader798: metadata journal open: replayFrom=1 applyTo=0 05:40:40,727 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader798: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 05:40:40,727 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - leader798: journal open: applyTo=0 05:40:40,728 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: journal writer started 05:40:40,728 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: waiting to receive commands 05:40:40,729 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest12029400409825354904/leader798/TermInfo.properties does not exist 05:40:40,729 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower800: Set currentTerm=0, votedFor=null 05:40:40,729 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower800: Local TermInfo store seeded with TermInfo{term=0} 05:40:40,729 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader798: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest12029400409825354904/leader798/journal-v1-1.log 05:40:40,729 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower800: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6756dbb2 05:40:40,729 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader798: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest12029400409825354904/leader798/metadata-v1-1.log 05:40:40,729 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower800: Moving last applied index from -1 to -1 05:40:40,729 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader798: no eligible files found 05:40:40,729 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader798: Set currentTerm=5, votedFor=leader798 05:40:40,729 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader798: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 05:40:40,729 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower800: empty journal: appending 0 entries 05:40:40,729 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower800: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:40,729 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower800: journal recovery completedin 170.5 μs with journalIndex=1 05:40:40,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader798: Set currentTerm=5, votedFor=leader798 05:40:40,731 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader798: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader798} 05:40:40,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader798: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@4fd51dd7 05:40:40,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader798: Moving last applied index from -1 to -1 05:40:40,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader798: empty journal: appending 0 entries 05:40:40,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader798: updated meta to JournalMeta[replayFrom=1, applyTo=0] 05:40:40,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader798: journal recovery completedin 181.0 μs with journalIndex=1 05:40:41,223 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 05:40:41,223 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader798 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.164:17355, Removed) 05:40:41,223 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader798 (Follower): Received ElectionTimeout - switching to Candidate 05:40:41,223 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader798 (Candidate): Election: Candidate has following voting peers: [follower799, follower800] 05:40:41,227 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader798: Set currentTerm=6, votedFor=leader798 05:40:41,227 PM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader798 (Candidate): Starting new election term 6 05:40:41,227 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader798 (Candidate): Sending RequestVote{term=6, candidateId=leader798, lastLogIndex=-1, lastLogTerm=-1} to peer follower799 05:40:41,227 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader798 (Candidate): Sending RequestVote{term=6, candidateId=leader798, lastLogIndex=-1, lastLogTerm=-1} to peer follower800 05:40:41,227 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower800 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 05:40:41,227 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower799 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 05:40:41,227 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower800 (Follower): Cluster state: [] 05:40:41,227 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower800 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader798, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 05:40:41,227 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower800 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader798, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 05:40:41,227 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower799 (Follower): Cluster state: [] 05:40:41,227 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower799 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader798, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 05:40:41,227 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower799 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader798, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 05:40:41,228 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader798 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 05:40:41,230 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower800: Set currentTerm=6, votedFor=null 05:40:41,230 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower800 (Follower): In requestVote: RequestVote{term=6, candidateId=leader798, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 05:40:41,231 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower799: Set currentTerm=6, votedFor=null 05:40:41,231 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower799 (Follower): In requestVote: RequestVote{term=6, candidateId=leader798, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 05:40:41,232 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower800: Set currentTerm=6, votedFor=leader798 05:40:41,232 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower800 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 05:40:41,232 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader798 (Candidate): RaftRPC message received RequestVoteReply{term=6, voteGranted=true}, my term is 6 05:40:41,232 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader798 (Candidate): handleRequestVoteReply: RequestVoteReply{term=6, voteGranted=true}, current voteCount: 1 05:40:41,233 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader798 (Leader): Election: Leader has following peers: [follower799, follower800] 05:40:41,233 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower799: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:41,233 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower800: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:41,233 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader798 (Candidate) :- Switching from behavior Candidate to Leader, election term: 6 05:40:41,233 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower800: Last sync leader does not match current leader leader798, need to catch up to -1 05:40:41,233 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader798 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower800 updated: matchIndex: -1, nextIndex: 0 05:40:41,233 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower800: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 17:40:41.233] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower800] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower800, initialSyncDone=false] 05:40:41,233 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower799: Set currentTerm=6, votedFor=leader798 05:40:41,233 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower799 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 05:40:41,233 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Last sync leader does not match current leader leader798, need to catch up to -1 05:40:41,233 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower799 updated: matchIndex: -1, nextIndex: 0 [WARN] [07/01/2025 17:40:41.233] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower799] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower799, initialSyncDone=false] 05:40:41,234 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower799: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 05:40:41,235 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - Leader created and elected 05:40:41,235 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader798: Persist data index=0 term=6 command=zero 05:40:41,235 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader798: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:41,235 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 05:40:41,235 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower799: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:41,235 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): sendAppendEntries: 0 is present for follower follower799 05:40:41,235 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: received 1 commands after 507.5 ms 05:40:41,235 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): Sending AppendEntries to follower follower799: AppendEntries{term=6, leaderId=leader798, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:41,235 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower800: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:41,235 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): sendAppendEntries: 0 is present for follower follower800 05:40:41,235 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower799 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader798, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:41,235 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower799 (Follower): Number of entries to be appended = 1 05:40:41,235 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower799 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 05:40:41,235 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): Sending AppendEntries to follower follower800: AppendEntries{term=6, leaderId=leader798, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:41,235 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower799 (Follower): Append entry to log zero 05:40:41,235 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower799: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:41,235 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader798: awaiting more completions to resolve 1 synchronized callback(s) 05:40:41,235 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower800 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader798, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 05:40:41,235 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower800 (Follower): Number of entries to be appended = 1 05:40:41,235 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower800 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 05:40:41,235 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower800 (Follower): Append entry to log zero 05:40:41,235 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower800: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:41,236 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower800 (Follower): Log size is now 1 05:40:41,236 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower800 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower800, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:41,236 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower800: Lagging 0 entries behind leader leader798 and reached -1 (of expected -1) 05:40:41,236 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: received 1 commands after 511.4 ms [WARN] [07/01/2025 17:40:41.236] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower800] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower800, initialSyncDone=true] 05:40:41,236 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: received 1 commands after 513.9 ms 05:40:41,236 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower799 (Follower): Log size is now 1 05:40:41,236 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower799 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower799, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:41,236 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached -1 (of expected -1) [WARN] [07/01/2025 17:40:41.236] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower799] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower799, initialSyncDone=true] 05:40:41,239 PM [leader798-writer-174] [DEBUG] RaftStorageCompleter - leader798: 1 completion(s) pending 05:40:41,239 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: completed 1 commands in 4.000 ms 05:40:41,239 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: waiting to receive commands 05:40:41,239 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader798: running 1 completion(s) 05:40:41,239 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader798: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=true} 05:40:41,239 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader798: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@6b23fc2a 05:40:41,239 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower800 updated: matchIndex: 0, nextIndex: 1 05:40:41,239 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 05:40:41,239 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader798 (Leader): Setting last applied to 0 05:40:41,239 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader798: Moving last applied index from -1 to 0 05:40:41,239 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader798: Applying state for log index 0 data zero 05:40:41,240 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader798: applyState called: zero 05:40:41,240 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: received 1 commands after 547.3 μs 05:40:41,240 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower800: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:41,240 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower799 updated: matchIndex: 0, nextIndex: 1 05:40:41,240 PM [leader798-writer-174] [DEBUG] EntryJournalV1 - leader798: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:41,240 PM [leader798-writer-174] [DEBUG] RaftStorageCompleter - leader798: 1 completion(s) pending 05:40:41,240 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower799: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 05:40:41,240 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: completed 1 commands in 115.2 μs 05:40:41,240 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: waiting to receive commands 05:40:41,240 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader798: running 1 completion(s) 05:40:41,240 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower799 (Follower): Commit index set to 0 05:40:41,240 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 0 (of expected -1) 05:40:41,240 PM [follower799-writer-172] [DEBUG] RaftStorageCompleter - follower799: 1 completion(s) pending 05:40:41,240 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower799 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 05:40:41,240 PM [follower800-writer-173] [DEBUG] RaftStorageCompleter - follower800: 1 completion(s) pending 05:40:41,240 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: completed 1 commands in 4.478 ms 05:40:41,240 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: waiting to receive commands 05:40:41,240 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: completed 1 commands in 4.308 ms 05:40:41,240 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: waiting to receive commands 05:40:41,240 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower799 (Follower): Setting last applied to 0 05:40:41,240 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower799: Moving last applied index from -1 to 0 05:40:41,240 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower799: Applying state for log index 0 data zero 05:40:41,240 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower799: applyState called: zero 05:40:41,240 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower799: running 1 completion(s) 05:40:41,240 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: received 1 commands after 181.3 μs 05:40:41,240 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower799: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:41,240 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower800: running 1 completion(s) 05:40:41,240 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower800: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 05:40:41,240 PM [follower799-writer-172] [DEBUG] EntryJournalV1 - follower799: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:41,240 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower800 (Follower): Commit index set to 0 05:40:41,240 PM [follower799-writer-172] [DEBUG] RaftStorageCompleter - follower799: 1 completion(s) pending 05:40:41,240 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower800: Lagging 0 entries behind leader leader798 and reached 0 (of expected -1) 05:40:41,240 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower800 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 05:40:41,240 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: completed 1 commands in 128.6 μs 05:40:41,241 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower800 (Follower): Setting last applied to 0 05:40:41,241 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: waiting to receive commands 05:40:41,241 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower800: Moving last applied index from -1 to 0 05:40:41,241 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower800: Applying state for log index 0 data zero 05:40:41,241 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower800: applyState called: zero 05:40:41,241 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower799: running 1 completion(s) 05:40:41,241 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: received 1 commands after 488.5 μs 05:40:41,241 PM [follower800-writer-173] [DEBUG] EntryJournalV1 - follower800: updated meta to JournalMeta[replayFrom=1, applyTo=1] 05:40:41,241 PM [follower800-writer-173] [DEBUG] RaftStorageCompleter - follower800: 1 completion(s) pending 05:40:41,241 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower800: running 1 completion(s) 05:40:41,241 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: completed 1 commands in 94.27 μs 05:40:41,241 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: waiting to receive commands 05:40:41,352 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower800: Lagging 0 entries behind leader leader798 and reached 0 (of expected -1) 05:40:41,353 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 0 (of expected -1) 05:40:41,473 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 0 (of expected -1) 05:40:41,593 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 0 (of expected -1) 05:40:41,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 0 (of expected -1) 05:40:41,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader798: Persist data index=1 term=6 command=one 05:40:41,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader798: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:41,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader798 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 05:40:41,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower799: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:41,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader798 (Leader): sendAppendEntries: 1 is present for follower follower799 05:40:41,793 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: received 1 commands after 553.3 ms 05:40:41,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader798 (Leader): Sending AppendEntries to follower follower799: AppendEntries{term=6, leaderId=leader798, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:41,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower800: active: false, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 05:40:41,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader798: awaiting more completions to resolve 1 synchronized callback(s) 05:40:41,793 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower799 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader798, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 05:40:41,794 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower799 (Follower): Number of entries to be appended = 1 05:40:41,794 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower799 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 05:40:41,794 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower799 (Follower): Append entry to log one 05:40:41,794 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower799: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:41,794 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower799 (Follower): Log size is now 2 05:40:41,794 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower799 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower799, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 05:40:41,794 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 0 (of expected -1) 05:40:41,794 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: received 1 commands after 553.2 ms 05:40:41,796 PM [follower799-writer-172] [DEBUG] RaftStorageCompleter - follower799: 1 completion(s) pending 05:40:41,796 PM [leader798-writer-174] [DEBUG] RaftStorageCompleter - leader798: 1 completion(s) pending 05:40:41,796 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: completed 1 commands in 2.447 ms 05:40:41,796 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: completed 1 commands in 1.911 ms 05:40:41,796 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: waiting to receive commands 05:40:41,796 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: waiting to receive commands 05:40:41,796 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader798: running 1 completion(s) 05:40:41,796 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower799: running 1 completion(s) 05:40:41,796 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader798: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=true} 05:40:41,796 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower799: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 05:40:41,796 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader798: completed synchronized callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0811ba8@47bc43da 05:40:41,796 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader798 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower799 updated: matchIndex: 1, nextIndex: 2 05:40:41,796 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader798 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 05:40:41,796 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader798 (Leader): Setting last applied to 1 05:40:41,796 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader798: Moving last applied index from 0 to 1 05:40:41,796 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader798: Applying state for log index 1 data one 05:40:41,796 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader798: applyState called: one 05:40:41,796 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: received 1 commands after 617.5 μs 05:40:41,797 PM [leader798-writer-174] [DEBUG] EntryJournalV1 - leader798: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:41,797 PM [leader798-writer-174] [DEBUG] RaftStorageCompleter - leader798: 1 completion(s) pending 05:40:41,797 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: completed 1 commands in 197.8 μs 05:40:41,797 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: waiting to receive commands 05:40:41,797 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader798: fakeSnapshot purging log to 0 for term 6 05:40:41,797 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader798 (Leader): Checking sendAppendEntries for follower follower799: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 05:40:41,797 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader798: running 1 completion(s) 05:40:41,797 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower799 (Follower): Commit index set to 1 05:40:41,797 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:41,797 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower799 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 05:40:41,797 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower799 (Follower): Setting last applied to 1 05:40:41,797 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower799: Moving last applied index from 0 to 1 05:40:41,797 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower799: Applying state for log index 1 data one 05:40:41,797 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower799: applyState called: one 05:40:41,797 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SnapshotManager - follower799: fakeSnapshot purging log to 0 for term 6 05:40:41,797 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: received 1 commands after 1.688 ms 05:40:41,798 PM [follower799-writer-172] [DEBUG] EntryJournalV1 - follower799: updated meta to JournalMeta[replayFrom=1, applyTo=2] 05:40:41,798 PM [follower799-writer-172] [DEBUG] RaftStorageCompleter - follower799: 1 completion(s) pending 05:40:41,798 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: completed 1 commands in 141.4 μs 05:40:41,798 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower799: running 1 completion(s) 05:40:41,798 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: waiting to receive commands 05:40:41,952 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:42,72 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:42,193 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:42,313 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:42,433 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:42,552 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:42,673 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:42,792 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:42,912 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:43,33 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:43,152 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:43,273 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:43,392 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:43,513 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:43,633 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:43,752 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:43,873 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:43,992 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:44,112 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:44,232 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:44,353 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:44,473 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:44,593 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:44,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:44,833 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:44,952 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:45,72 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:45,192 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:45,313 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:45,433 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:45,552 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:45,673 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:45,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:45,913 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:46,32 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:46,153 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:46,273 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:46,393 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:46,513 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:46,632 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:46,753 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower799: Lagging 0 entries behind leader leader798 and reached 1 (of expected -1) 05:40:46,844 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower799-collector801#-210929345] 05:40:46,855 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower799] 05:40:46,855 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower799: terminating thread VirtualThread[#960,follower799-writer-172]/waiting 05:40:46,855 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: received 1 commands after 5.058 s 05:40:46,855 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: completed 1 commands in 16.51 μs 05:40:46,856 PM [follower799-writer-172] [DEBUG] JournalWriteTask - follower799: journal writer stopped 05:40:46,856 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower799: journal closed 05:40:46,856 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower799: journal closed 05:40:46,856 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower799: stopped executor with 0 remaining tasks 05:40:46,856 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower799: stopped executor with 0 remaining tasks 05:40:46,856 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower800-collector802#571755978] 05:40:46,856 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower800] 05:40:46,856 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower800: terminating thread VirtualThread[#961,follower800-writer-173]/waiting 05:40:46,856 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: received 1 commands after 5.616 s 05:40:46,857 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: completed 1 commands in 10.12 μs 05:40:46,857 PM [follower800-writer-173] [DEBUG] JournalWriteTask - follower800: journal writer stopped 05:40:46,857 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower800: journal closed 05:40:46,857 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower800: journal closed 05:40:46,857 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower800: stopped executor with 0 remaining tasks 05:40:46,857 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower800: stopped executor with 0 remaining tasks 05:40:46,857 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader798-collector803#-172610288] 05:40:46,857 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader798] 05:40:46,857 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - leader798: terminating thread VirtualThread[#962,leader798-writer-174]/waiting 05:40:46,857 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: received 1 commands after 5.061 s 05:40:46,857 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: completed 1 commands in 8.930 μs 05:40:46,857 PM [leader798-writer-174] [DEBUG] JournalWriteTask - leader798: journal writer stopped 05:40:46,857 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader798: journal closed 05:40:46,857 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader798: journal closed 05:40:46,857 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader798: stopped executor with 0 remaining tasks 05:40:46,857 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader798: stopped executor with 0 remaining tasks [INFO] [07/01/2025 17:40:46.861] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 05:40:46,862 PM [test-pekko.actor.default-dispatcher-17] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [07/01/2025 17:40:46.862] [test-pekko.actor.default-dispatcher-12] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.171.164:17355) [INFO] [07/01/2025 17:40:46.863] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.164:17355] - Exiting completed 05:40:46,863 PM [test-pekko.actor.default-dispatcher-17] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.164:17355] - Exiting completed [INFO] [07/01/2025 17:40:46.863] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.164:17355] - Shutting down... 05:40:46,863 PM [test-pekko.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.164:17355] - Shutting down... [INFO] [07/01/2025 17:40:46.863] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.164:17355] - Successfully shut down 05:40:46,863 PM [test-pekko.actor.default-dispatcher-17] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.164:17355] - Successfully shut down [INFO] [07/01/2025 17:40:46.865] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.171.164:17355/system/remoting-terminator] Shutting down remote daemon. 05:40:46,865 PM [test-pekko.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 05:40:46,865 PM [test-pekko.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [07/01/2025 17:40:46.865] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.171.164:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 05:40:46,867 PM [test-pekko.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [INFO] [07/01/2025 17:40:46.867] [test-pekko.remote.default-remote-dispatcher-7] [pekko://test@10.30.171.164:17355/system/remoting-terminator] Remoting shut down. [DEBUG] [07/01/2025 17:40:46.868] [test-pekko.actor.internal-dispatcher-2] [EventStream] shutting down: StandardOutLogger