[DEBUG] [08/11/2025 16:02:59.000] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 04:02:59,0 PM [test-pekko.actor.default-dispatcher-6] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [08/11/2025 16:02:59.000] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [08/11/2025 16:02:59.000] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [08/11/2025 16:02:59.015] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.10:17355] with UID [-2780174061687192342] [INFO] [08/11/2025 16:02:59.016] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.10:17355] - Starting up, Pekko version [1.0.3] ... 04:02:59,17 PM [test-pekko.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.10:17355] with UID [-2780174061687192342] 04:02:59,17 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.10:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [08/11/2025 16:02:59.018] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.10:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [08/11/2025 16:02:59.018] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.10:17355] - Started up successfully 04:02:59,18 PM [test-pekko.actor.default-dispatcher-7] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.10:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 04:02:59,18 PM [test-pekko.actor.default-dispatcher-7] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.10:17355] - Started up successfully 04:02:59,22 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.10:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 04:02:59,22 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.10:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining [INFO] [08/11/2025 16:02:59.018] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.10:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [08/11/2025 16:02:59.018] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.10:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 04:02:59,27 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testReplicationsWithLaggingFollowerCaughtUpViaAppendEntries starting: sending 2 new payloads [WARN] [08/11/2025 16:02:59.031] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower732]: org.opendaylight.controller.cluster.common.actor.Monitor@1bd1ab4e 04:02:59,32 PM [test-pekko.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower732] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][08/11/2025 16:02:59.032] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower732] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,35 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower732: Starting recovery with journal batch size 1000 04:02:59,36 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower732: started executor 04:02:59,36 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower732: started executor 04:02:59,37 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower732: metadata journal open: replayFrom=1 applyTo=0 04:02:59,38 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower732: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:02:59,38 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - follower732: journal open: applyTo=0 04:02:59,38 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: journal writer started 04:02:59,38 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: waiting to receive commands [WARN] [08/11/2025 16:02:59.038] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower733]: org.opendaylight.controller.cluster.common.actor.Monitor@2ab6fa93 [INFO] [pekkoDeadLetter][08/11/2025 16:02:59.038] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower733] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,39 PM [test-pekko.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower733] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,39 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower733: Starting recovery with journal batch size 1000 04:02:59,39 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower733: started executor 04:02:59,39 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower733: started executor 04:02:59,39 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower733: metadata journal open: replayFrom=1 applyTo=0 04:02:59,40 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower733: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:02:59,40 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - follower733: journal open: applyTo=0 04:02:59,40 PM [follower733-writer-133] [DEBUG] JournalWriteTask - follower733: journal writer started 04:02:59,40 PM [follower733-writer-133] [DEBUG] JournalWriteTask - follower733: waiting to receive commands 04:02:59,43 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/follower732/TermInfo.properties does not exist 04:02:59,43 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower732: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/follower732/journal-v1-1.log 04:02:59,43 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower732: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/follower732/metadata-v1-1.log 04:02:59,43 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower732: no eligible files found 04:02:59,43 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/follower733/TermInfo.properties does not exist 04:02:59,43 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower732: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:02:59,43 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower733: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/follower733/journal-v1-1.log 04:02:59,43 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower733: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/follower733/metadata-v1-1.log 04:02:59,43 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower733: no eligible files found 04:02:59,43 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower733: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:02:59,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower732: Set currentTerm=0, votedFor=null 04:02:59,47 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower732: Local TermInfo store seeded with TermInfo{term=0} 04:02:59,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower732: Pekko recovery completed and can restore from snapshot 04:02:59,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower732: starting journal recovery 04:02:59,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower732: Moving last applied index from -1 to -1 04:02:59,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower732: empty journal: appending 0 entries 04:02:59,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower732: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:02:59,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower732: journal recovery completedin 198.7 μs with journalIndex=1 04:02:59,47 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower732: Journal recovery completed and can restore from snapshot 04:02:59,48 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower733: Set currentTerm=0, votedFor=null 04:02:59,48 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower733: Local TermInfo store seeded with TermInfo{term=0} 04:02:59,48 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower733: Pekko recovery completed and can restore from snapshot 04:02:59,48 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower733: starting journal recovery 04:02:59,48 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower733: Moving last applied index from -1 to -1 04:02:59,48 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower733: empty journal: appending 0 entries 04:02:59,48 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower733: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:02:59,48 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower733: journal recovery completedin 269.0 μs with journalIndex=1 04:02:59,48 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower733: Journal recovery completed and can restore from snapshot [WARN] [08/11/2025 16:02:59.140] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader731]: org.opendaylight.controller.cluster.common.actor.Monitor@62069fcd [INFO] [pekkoDeadLetter][08/11/2025 16:02:59.140] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader731] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,141 PM [test-pekko.actor.default-dispatcher-7] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader731] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,142 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - leader731: Starting recovery with journal batch size 1000 04:02:59,142 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader731: started executor 04:02:59,142 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader731: started executor 04:02:59,142 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader731: metadata journal open: replayFrom=1 applyTo=0 04:02:59,143 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader731: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:02:59,143 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - leader731: journal open: applyTo=0 04:02:59,143 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: journal writer started 04:02:59,143 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: waiting to receive commands 04:02:59,145 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/leader731/TermInfo.properties does not exist 04:02:59,145 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader731: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/leader731/journal-v1-1.log 04:02:59,145 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader731: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/leader731/metadata-v1-1.log 04:02:59,145 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader731: no eligible files found 04:02:59,145 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader731: Set currentTerm=5, votedFor=leader731 04:02:59,145 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader731: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:02:59,149 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader731: Set currentTerm=5, votedFor=leader731 04:02:59,149 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader731: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader731} 04:02:59,149 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader731: taking snapshot to clear Pekko persistence to 1 04:02:59,149 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Recovery - leader731: no applied entries in recovery log, re-snapshotting EntryInfo[index=-1, term=-1] 04:02:59,149 PM [test-pekko.actor.default-dispatcher-6] [INFO] Recovery - leader731: Taking snapshot on entry with index -1 04:02:59,149 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader731: support() called 04:02:59,149 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader731: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/leader731/snapshot-00000000689A1433-08EF7FA5.tmp 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader731: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/leader731/snapshot-00000000689A1433-08EF7FA5.v1 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader731: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/leader731/snapshot-00000000689A1433-08EF7FA5.v1 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader731: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/leader731/TermInfo.properties 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader731: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/leader731/journal-v1-1.log 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader731: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/leader731/metadata-v1-1.log 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader731: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:02:59.149913509Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest6167207543973026994/leader731/snapshot-00000000689A1433-08EF7FA5.v1, size=221, entryCompress=NONE, stateCompress=NONE} 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [INFO] Recovery - leader731: Snapshot completed in 1.915 ms, resetting timer for the next recovery snapshot 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader731: Pekko recovery completed and cannot restore from snapshot 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader731: starting journal recovery 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader731: Moving last applied index from -1 to -1 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader731: empty journal: appending 0 entries 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader731: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader731: journal recovery completedin 208.9 μs with journalIndex=1 04:02:59,151 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader731: Journal recovery completed and cannot restore from snapshot 04:02:59,152 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader731: ignoring DeleteMessagesSuccess(1) 04:02:59,593 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 04:02:59,594 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - leader731 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.10:17355, Removed) 04:02:59,594 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - leader731 (Follower): Received ElectionTimeout - switching to Candidate 04:02:59,594 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader731 (Candidate): Election: Candidate has following voting peers: [follower733, follower732] 04:02:59,597 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader731: Set currentTerm=6, votedFor=leader731 04:02:59,597 PM [test-pekko.actor.default-dispatcher-11] [INFO] Candidate - leader731 (Candidate): Starting new election term 6 04:02:59,597 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader731 (Candidate): Sending RequestVote{term=6, candidateId=leader731, lastLogIndex=-1, lastLogTerm=-1} to peer follower733 04:02:59,597 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader731 (Candidate): Sending RequestVote{term=6, candidateId=leader731, lastLogIndex=-1, lastLogTerm=-1} to peer follower732 04:02:59,597 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader731 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 04:02:59,597 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower733 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 04:02:59,597 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower733 (Follower): Cluster state: [] 04:02:59,597 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower733 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader731, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 04:02:59,597 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower732 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 04:02:59,597 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower733 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader731, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 04:02:59,598 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower732 (Follower): Cluster state: [] 04:02:59,598 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower732 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader731, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 04:02:59,598 PM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower732 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader731, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 04:02:59,600 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower733: Set currentTerm=6, votedFor=null 04:02:59,600 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower733 (Follower): In requestVote: RequestVote{term=6, candidateId=leader731, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 04:02:59,602 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower732: Set currentTerm=6, votedFor=null 04:02:59,602 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower732 (Follower): In requestVote: RequestVote{term=6, candidateId=leader731, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 04:02:59,602 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower733: Set currentTerm=6, votedFor=leader731 04:02:59,602 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower733 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 04:02:59,602 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - leader731 (Candidate): RaftRPC message received RequestVoteReply{term=6, voteGranted=true}, my term is 6 04:02:59,602 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - leader731 (Candidate): handleRequestVoteReply: RequestVoteReply{term=6, voteGranted=true}, current voteCount: 1 04:02:59,603 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader731 (Leader): Election: Leader has following peers: [follower733, follower732] 04:02:59,603 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower733: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:02:59,603 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower732: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:02:59,603 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActorBehavior - leader731 (Candidate) :- Switching from behavior Candidate to Leader, election term: 6 04:02:59,603 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower733: Last sync leader does not match current leader leader731, need to catch up to -1 04:02:59,603 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader731 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower733 updated: matchIndex: -1, nextIndex: 0 04:02:59,603 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower733: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/11/2025 16:02:59.603] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower733] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower733, initialSyncDone=false] [INFO] [pekkoDeadLetter][08/11/2025 16:02:59.603] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower733] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower733] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,603 PM [test-pekko.actor.default-dispatcher-11] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower733] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,605 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower732: Set currentTerm=6, votedFor=leader731 04:02:59,605 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower732 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 04:02:59,606 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower732: Last sync leader does not match current leader leader731, need to catch up to -1 04:02:59,606 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader731 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower732 updated: matchIndex: -1, nextIndex: 0 04:02:59,606 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower732: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/11/2025 16:02:59.606] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower732] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower732, initialSyncDone=false] [INFO] [pekkoDeadLetter][08/11/2025 16:02:59.606] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower732] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower732] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,606 PM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower732] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,608 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - Leader created and elected 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader731: Persist data index=0 term=6 command=zero 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader731: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower733: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): sendAppendEntries: 0 is present for follower follower733 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): Sending AppendEntries to follower follower733: AppendEntries{term=6, leaderId=leader731, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower732: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): sendAppendEntries: 0 is present for follower follower732 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): Sending AppendEntries to follower follower732: AppendEntries{term=6, leaderId=leader731, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader731: Persist data index=1 term=6 command=one 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader731: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower733: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): sendAppendEntries: 0 is present for follower follower733 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower732: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:02:59,608 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader731, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): sendAppendEntries: 0 is present for follower follower732 04:02:59,608 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): Number of entries to be appended = 1 04:02:59,608 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 04:02:59,608 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: received 2 commands after 465.2 ms 04:02:59,608 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): Append entry to log zero 04:02:59,608 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower732: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:02:59,608 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): Log size is now 1 04:02:59,608 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower732, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:02:59,608 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower732: Lagging 0 entries behind leader leader731 and reached -1 (of expected -1) 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower732 updated: matchIndex: 0, nextIndex: 1 04:02:59,608 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower732: awaiting more completions to resolve 1 synchronized callback(s) 04:02:59,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower732: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:02:59,609 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): sendAppendEntries: 1 is present for follower follower732 04:02:59,609 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader731 (Leader): Sending AppendEntries to follower follower732: AppendEntries{term=6, leaderId=leader731, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:02:59,609 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: received 1 commands after 571.0 ms 04:02:59,612 PM [leader731-writer-134] [DEBUG] RaftStorageCompleter - leader731: 2 completion(s) pending 04:02:59,612 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: completed 2 commands in 3.791 ms 04:02:59,612 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: waiting to receive commands 04:02:59,612 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader731: running 2 completion(s) 04:02:59,612 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader731: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:02:59,612 PM [follower732-writer-132] [DEBUG] RaftStorageCompleter - follower732: 1 completion(s) pending 04:02:59,612 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader731 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 04:02:59,612 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: completed 1 commands in 3.614 ms 04:02:59,612 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower732: running 1 completion(s) 04:02:59,612 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: waiting to receive commands 04:02:59,612 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader731 (Leader): Setting last applied to 0 04:02:59,612 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader731: Moving last applied index from -1 to 0 04:02:59,612 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower732: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:02:59,612 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower732: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=6, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@35f0d73} 04:02:59,612 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader731: Applying state for log index 0 data zero 04:02:59,612 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader731: applyState called: zero 04:02:59,612 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower732: synchronized callbacks after 3.900 ms 04:02:59,612 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader731, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:02:59,613 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader731: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): Number of entries to be appended = 1 04:02:59,613 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: received 1 commands after 378.2 μs 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): Append entry to log one 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower732: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): Log size is now 2 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower732, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower732: Lagging 0 entries behind leader leader731 and reached -1 (of expected -1) 04:02:59,613 PM [leader731-writer-134] [DEBUG] EntryJournalV1 - leader731: updated meta to JournalMeta[replayFrom=1, applyTo=1] 04:02:59,613 PM [leader731-writer-134] [DEBUG] RaftStorageCompleter - leader731: 1 completion(s) pending 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower732: awaiting more completions to resolve 1 synchronized callback(s) 04:02:59,613 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: completed 1 commands in 164.0 μs 04:02:59,613 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: waiting to receive commands 04:02:59,613 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader731: running 1 completion(s) 04:02:59,613 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: received 1 commands after 418.9 μs 04:02:59,613 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader731 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower732 updated: matchIndex: 1, nextIndex: 2 04:02:59,613 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader731 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 04:02:59,613 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader731 (Leader): Setting last applied to 1 04:02:59,613 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader731: Moving last applied index from 0 to 1 04:02:59,613 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader731: Applying state for log index 1 data one 04:02:59,613 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader731: applyState called: one 04:02:59,613 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower732: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:02:59,613 PM [follower732-writer-132] [DEBUG] RaftStorageCompleter - follower732: 1 completion(s) pending 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower732: running 1 completion(s) 04:02:59,613 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: completed 1 commands in 527.4 μs 04:02:59,613 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: waiting to receive commands 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower732: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:02:59,613 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: received 1 commands after 564.9 μs 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower732: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=6, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@6394d825} 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower732: synchronized callbacks after 651.6 μs 04:02:59,613 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): Commit index set to 1 04:02:59,613 PM [leader731-writer-134] [DEBUG] EntryJournalV1 - leader731: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower732: Lagging 0 entries behind leader leader731 and reached 1 (of expected -1) 04:02:59,614 PM [leader731-writer-134] [DEBUG] RaftStorageCompleter - leader731: 1 completion(s) pending 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower732 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 1 04:02:59,614 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: completed 1 commands in 222.0 μs 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower732 (Follower): Setting last applied to 0 04:02:59,614 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: waiting to receive commands 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower732: Moving last applied index from -1 to 0 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower732: Applying state for log index 0 data zero 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower732: applyState called: zero 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower732 (Follower): Setting last applied to 1 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower732: Moving last applied index from 0 to 1 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower732: Applying state for log index 1 data one 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower732: applyState called: one 04:02:59,614 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: received 1 commands after 520.5 μs 04:02:59,614 PM [follower732-writer-132] [DEBUG] EntryJournalV1 - follower732: updated meta to JournalMeta[replayFrom=1, applyTo=2] [WARN] [08/11/2025 16:02:59.614] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower732] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower732, initialSyncDone=true] 04:02:59,614 PM [follower732-writer-132] [DEBUG] RaftStorageCompleter - follower732: 1 completion(s) pending 04:02:59,614 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower732: running 1 completion(s) 04:02:59,614 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: completed 1 commands in 145.3 μs 04:02:59,614 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: waiting to receive commands [INFO] [pekkoDeadLetter][08/11/2025 16:02:59.614] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower732] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower732] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,614 PM [test-pekko.actor.default-dispatcher-7] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower732] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,614 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader731: running 1 completion(s) 04:02:59,713 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader731 (Leader): sendAppendEntries: 0 is present for follower follower733 04:02:59,713 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader731 (Leader): Sending AppendEntries to follower follower733: AppendEntries{term=6, leaderId=leader731, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=false}, JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 04:02:59,833 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader731 (Leader): sendAppendEntries: 0 is present for follower follower733 04:02:59,833 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader731 (Leader): Sending AppendEntries to follower follower733: AppendEntries{term=6, leaderId=leader731, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=false}, JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 04:02:59,833 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower732: Lagging 0 entries behind leader leader731 and reached 1 (of expected -1) 04:02:59,859 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testReplicationsWithLaggingFollowerCaughtUpViaAppendEntries: new entries applied - resuming follower follower733 04:02:59,953 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader731 (Leader): sendAppendEntries: 0 is present for follower follower733 04:02:59,954 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader731 (Leader): Sending AppendEntries to follower follower733: AppendEntries{term=6, leaderId=leader731, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=false}, JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower733 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader731, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=false}, JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower733 (Follower): Number of entries to be appended = 2 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower733 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower733 (Follower): Append entry to log zero 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower733: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:02:59,954 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower732: Lagging 0 entries behind leader leader731 and reached 1 (of expected -1) 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower733 (Follower): Append entry to log one 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower733: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower733 (Follower): Log size is now 2 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower733 (Follower): Commit index set to 1 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower733 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower733, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower733: Lagging 0 entries behind leader leader731 and reached 1 (of expected -1) 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower733 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 1 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower733 (Follower): Setting last applied to 0 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower733: Moving last applied index from -1 to 0 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower733: Applying state for log index 0 data zero 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower733: applyState called: zero 04:02:59,954 PM [follower733-writer-133] [DEBUG] JournalWriteTask - follower733: received 2 commands after 914.4 ms 04:02:59,954 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader731 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower733 updated: matchIndex: 1, nextIndex: 2 04:02:59,954 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - leader731: fakeSnapshot purging log to 0 for term 6 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower733 (Follower): Setting last applied to 1 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower733: Moving last applied index from 0 to 1 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower733: Applying state for log index 1 data one 04:02:59,954 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader731 (Leader): Checking sendAppendEntries for follower follower733: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower733: applyState called: one 04:02:59,954 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower733: awaiting more completions to resolve 2 synchronized callback(s) 04:02:59,959 PM [follower733-writer-133] [DEBUG] RaftStorageCompleter - follower733: 2 completion(s) pending 04:02:59,959 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower733: running 2 completion(s) 04:02:59,959 PM [follower733-writer-133] [DEBUG] JournalWriteTask - follower733: completed 2 commands in 5.108 ms 04:02:59,959 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower733: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:02:59,959 PM [follower733-writer-133] [DEBUG] JournalWriteTask - follower733: received 1 commands after 6.850 μs 04:02:59,959 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower733: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=6, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@6cb3d7df} 04:02:59,959 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower733: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:02:59,959 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower733: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=6, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@6cb3d7df} 04:02:59,959 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower733: synchronized callbacks after 5.081 ms 04:02:59,959 PM [follower733-writer-133] [DEBUG] EntryJournalV1 - follower733: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:02:59,959 PM [follower733-writer-133] [DEBUG] RaftStorageCompleter - follower733: 1 completion(s) pending 04:02:59,959 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower733: running 1 completion(s) 04:02:59,959 PM [follower733-writer-133] [DEBUG] JournalWriteTask - follower733: completed 1 commands in 207.2 μs 04:02:59,959 PM [follower733-writer-133] [DEBUG] JournalWriteTask - follower733: waiting to receive commands [WARN] [08/11/2025 16:02:59.960] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower733] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower733, initialSyncDone=true] [INFO] [pekkoDeadLetter][08/11/2025 16:02:59.960] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower733] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower733] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:02:59,960 PM [test-pekko.actor.default-dispatcher-15] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower733] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:03:00,74 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower733: Lagging 0 entries behind leader leader731 and reached 1 (of expected -1) 04:03:00,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower732: Lagging 0 entries behind leader leader731 and reached 1 (of expected -1) 04:03:00,74 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - follower733: fakeSnapshot purging log to 0 for term 6 04:03:00,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower732: fakeSnapshot purging log to 0 for term 6 04:03:00,111 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testReplicationsWithLaggingFollowerCaughtUpViaAppendEntries complete 04:03:00,111 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower732-collector734#130968596] 04:03:00,122 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower732] 04:03:00,122 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EnabledRaftStorage - follower732: terminating thread VirtualThread[#901,follower732-writer-132]/waiting 04:03:00,122 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: received 1 commands after 508.3 ms 04:03:00,122 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: completed 1 commands in 28.88 μs 04:03:00,122 PM [follower732-writer-132] [DEBUG] JournalWriteTask - follower732: journal writer stopped 04:03:00,123 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower732: journal closed 04:03:00,123 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - follower732: journal closed 04:03:00,123 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower732: stopped executor with 0 remaining tasks 04:03:00,123 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower732: stopped executor with 0 remaining tasks 04:03:00,123 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower733-collector735#1674240409] 04:03:00,123 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower733] 04:03:00,123 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EnabledRaftStorage - follower733: terminating thread VirtualThread[#902,follower733-writer-133]/waiting 04:03:00,123 PM [follower733-writer-133] [DEBUG] JournalWriteTask - follower733: received 1 commands after 163.8 ms 04:03:00,123 PM [follower733-writer-133] [DEBUG] JournalWriteTask - follower733: completed 1 commands in 7.970 μs 04:03:00,123 PM [follower733-writer-133] [DEBUG] JournalWriteTask - follower733: journal writer stopped 04:03:00,123 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower733: journal closed 04:03:00,123 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - follower733: journal closed 04:03:00,123 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower733: stopped executor with 0 remaining tasks 04:03:00,124 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower733: stopped executor with 0 remaining tasks 04:03:00,124 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader731-collector736#-87568459] 04:03:00,124 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader731] 04:03:00,124 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EnabledRaftStorage - leader731: terminating thread VirtualThread[#904,leader731-writer-134]/waiting 04:03:00,124 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: received 1 commands after 510.4 ms 04:03:00,124 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: completed 1 commands in 7.330 μs 04:03:00,124 PM [leader731-writer-134] [DEBUG] JournalWriteTask - leader731: journal writer stopped 04:03:00,124 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader731: journal closed 04:03:00,124 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - leader731: journal closed 04:03:00,124 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader731: stopped executor with 0 remaining tasks 04:03:00,124 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader731: stopped executor with 0 remaining tasks 04:03:00,125 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaInstallSnapshot starting [WARN] [08/11/2025 16:03:00.126] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower741]: org.opendaylight.controller.cluster.common.actor.Monitor@78b15b31 [INFO] [pekkoDeadLetter][08/11/2025 16:03:00.126] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower741] to Actor[pekko://test/user/termination-monitor] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:03:00,127 PM [test-pekko.actor.default-dispatcher-7] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower741] to Actor[pekko://test/user/termination-monitor] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:03:00,127 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower741: Starting recovery with journal batch size 1000 04:03:00,128 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower741: started executor 04:03:00,128 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower741: started executor 04:03:00,128 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower741: metadata journal open: replayFrom=1 applyTo=0 04:03:00,129 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower741: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:00,129 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - follower741: journal open: applyTo=0 04:03:00,129 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: journal writer started 04:03:00,129 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands [WARN] [08/11/2025 16:03:00.130] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower742]: org.opendaylight.controller.cluster.common.actor.Monitor@34595f6a 04:03:00,130 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower741/TermInfo.properties does not exist 04:03:00,130 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower741: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower741/journal-v1-1.log 04:03:00,130 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower741: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower741/metadata-v1-1.log 04:03:00,131 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower741: no eligible files found 04:03:00,131 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower741: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:00,131 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActor - follower742: Starting recovery with journal batch size 1000 [INFO] [pekkoDeadLetter][08/11/2025 16:03:00.131] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower742] to Actor[pekko://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:03:00,131 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower742: started executor 04:03:00,131 PM [test-pekko.actor.default-dispatcher-16] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower742] to Actor[pekko://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:03:00,131 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower742: started executor 04:03:00,133 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower742: metadata journal open: replayFrom=1 applyTo=0 04:03:00,133 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower742: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:00,133 PM [test-pekko.actor.default-dispatcher-7] [INFO] EnabledRaftStorage - follower742: journal open: applyTo=0 04:03:00,133 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: journal writer started 04:03:00,133 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: waiting to receive commands 04:03:00,134 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower741: Set currentTerm=0, votedFor=null 04:03:00,134 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower741: Local TermInfo store seeded with TermInfo{term=0} 04:03:00,134 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower741: Pekko recovery completed and can restore from snapshot 04:03:00,134 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower741: starting journal recovery 04:03:00,134 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower741: Moving last applied index from -1 to -1 04:03:00,134 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower741: empty journal: appending 0 entries 04:03:00,135 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower741: updated meta to JournalMeta[replayFrom=1, applyTo=0] [WARN] [08/11/2025 16:03:00.135] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader740]: org.opendaylight.controller.cluster.common.actor.Monitor@3c61ef14 04:03:00,135 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower741: journal recovery completedin 160.9 μs with journalIndex=1 04:03:00,135 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower741: Journal recovery completed and can restore from snapshot [INFO] [pekkoDeadLetter][08/11/2025 16:03:00.135] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader740] to Actor[pekko://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:03:00,135 PM [test-pekko.actor.default-dispatcher-16] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader740] to Actor[pekko://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 04:03:00,135 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActor - leader740: Starting recovery with journal batch size 1000 04:03:00,135 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader740: started executor 04:03:00,136 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader740: started executor 04:03:00,136 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - leader740: metadata journal open: replayFrom=1 applyTo=0 04:03:00,136 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - leader740: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:00,136 PM [test-pekko.actor.default-dispatcher-7] [INFO] EnabledRaftStorage - leader740: journal open: applyTo=0 04:03:00,137 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower742/TermInfo.properties does not exist 04:03:00,137 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower742: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower742/journal-v1-1.log 04:03:00,137 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower742: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower742/metadata-v1-1.log 04:03:00,137 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower742: no eligible files found 04:03:00,137 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower742: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:00,137 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/TermInfo.properties does not exist 04:03:00,137 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/journal-v1-1.log 04:03:00,137 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/metadata-v1-1.log 04:03:00,138 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader740: no eligible files found 04:03:00,138 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader740: Set currentTerm=5, votedFor=leader740 04:03:00,138 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - leader740: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:00,138 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: journal writer started 04:03:00,138 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:00,143 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower742: Set currentTerm=0, votedFor=null 04:03:00,143 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower742: Local TermInfo store seeded with TermInfo{term=0} 04:03:00,143 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower742: Pekko recovery completed and can restore from snapshot 04:03:00,143 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower742: starting journal recovery 04:03:00,143 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower742: Moving last applied index from -1 to -1 04:03:00,143 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower742: empty journal: appending 0 entries 04:03:00,143 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower742: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:00,143 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower742: journal recovery completedin 112.6 μs with journalIndex=1 04:03:00,143 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower742: Journal recovery completed and can restore from snapshot 04:03:00,143 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader740: Set currentTerm=5, votedFor=leader740 04:03:00,143 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - leader740: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader740} 04:03:00,143 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - leader740: taking snapshot to clear Pekko persistence to 1 04:03:00,143 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Recovery - leader740: no applied entries in recovery log, re-snapshotting EntryInfo[index=-1, term=-1] 04:03:00,143 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - leader740: Taking snapshot on entry with index -1 04:03:00,144 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader740: support() called 04:03:00,144 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader740: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1434-0899B446.tmp 04:03:00,146 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader740: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1434-0899B446.v1 04:03:00,147 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader740: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1434-0899B446.v1 04:03:00,147 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/TermInfo.properties 04:03:00,147 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/journal-v1-1.log 04:03:00,147 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/metadata-v1-1.log 04:03:00,147 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader740: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:00.144290886Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1434-0899B446.v1, size=221, entryCompress=NONE, stateCompress=NONE} 04:03:00,148 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - leader740: Snapshot completed in 4.269 ms, resetting timer for the next recovery snapshot 04:03:00,148 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader740: Pekko recovery completed and cannot restore from snapshot 04:03:00,148 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader740: starting journal recovery 04:03:00,148 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader740: Moving last applied index from -1 to -1 04:03:00,148 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - leader740: empty journal: appending 0 entries 04:03:00,148 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:00,148 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - leader740: journal recovery completedin 192.2 μs with journalIndex=1 04:03:00,148 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader740: Journal recovery completed and cannot restore from snapshot 04:03:00,158 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader740: ignoring DeleteMessagesSuccess(1) 04:03:00,663 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 04:03:00,663 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - leader740 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.10:17355, Removed) 04:03:00,664 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - leader740 (Follower): Received ElectionTimeout - switching to Candidate 04:03:00,664 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader740 (Candidate): Election: Candidate has following voting peers: [follower742, follower741] 04:03:00,668 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader740: Set currentTerm=6, votedFor=leader740 04:03:00,668 PM [test-pekko.actor.default-dispatcher-16] [INFO] Candidate - leader740 (Candidate): Starting new election term 6 04:03:00,668 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader740 (Candidate): Sending RequestVote{term=6, candidateId=leader740, lastLogIndex=-1, lastLogTerm=-1} to peer follower742 04:03:00,668 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader740 (Candidate): Sending RequestVote{term=6, candidateId=leader740, lastLogIndex=-1, lastLogTerm=-1} to peer follower741 04:03:00,668 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader740 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 04:03:00,668 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower742 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 04:03:00,668 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 04:03:00,669 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Cluster state: [] 04:03:00,669 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower742 (Follower): Cluster state: [] 04:03:00,669 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower742 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader740, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 04:03:00,669 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader740, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 04:03:00,669 PM [test-pekko.actor.default-dispatcher-7] [INFO] Follower - follower741 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader740, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 04:03:00,669 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower742 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader740, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 04:03:00,675 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower742: Set currentTerm=6, votedFor=null 04:03:00,675 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower741: Set currentTerm=6, votedFor=null 04:03:00,675 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower742 (Follower): In requestVote: RequestVote{term=6, candidateId=leader740, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 04:03:00,675 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): In requestVote: RequestVote{term=6, candidateId=leader740, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower742: Set currentTerm=6, votedFor=leader740 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower742 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader740 (Candidate): RaftRPC message received RequestVoteReply{term=6, voteGranted=true}, my term is 6 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader740 (Candidate): handleRequestVoteReply: RequestVoteReply{term=6, voteGranted=true}, current voteCount: 1 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Election: Leader has following peers: [follower742, follower741] 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader740 (Candidate) :- Switching from behavior Candidate to Leader, election term: 6 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower742: Last sync leader does not match current leader leader740, need to catch up to -1 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower742 updated: matchIndex: -1, nextIndex: 0 04:03:00,678 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/11/2025 16:03:00.679] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower742] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower742, initialSyncDone=false] 04:03:00,680 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower741: Set currentTerm=6, votedFor=leader740 04:03:00,680 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 04:03:00,680 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Last sync leader does not match current leader leader740, need to catch up to -1 04:03:00,680 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower741 updated: matchIndex: -1, nextIndex: 0 [WARN] [08/11/2025 16:03:00.680] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower741] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower741, initialSyncDone=false] 04:03:00,680 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:00,706 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - Leader created and elected 04:03:00,706 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader740: Persist data index=0 term=6 command=zero 04:03:00,706 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader740: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:00,706 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 04:03:00,706 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:00,706 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 0 is present for follower follower742 04:03:00,706 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 568.5 ms 04:03:00,706 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Sending AppendEntries to follower follower742: AppendEntries{term=6, leaderId=leader740, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 0 is present for follower follower741 04:03:00,707 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader740, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Sending AppendEntries to follower follower741: AppendEntries{term=6, leaderId=leader740, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:00,707 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): Number of entries to be appended = 1 04:03:00,707 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 04:03:00,707 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): Append entry to log zero 04:03:00,707 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower742: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader740: Persist data index=1 term=6 command=one 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader740: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:03:00,707 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): Log size is now 1 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 04:03:00,707 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower742, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:00,707 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower742: Lagging 0 entries behind leader leader740 and reached -1 (of expected -1) 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:00,707 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower742: awaiting more completions to resolve 1 synchronized callback(s) 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 0 is present for follower follower742 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 0 is present for follower follower741 04:03:00,707 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader740, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:00,707 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: received 1 commands after 573.5 ms 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower742 updated: matchIndex: 0, nextIndex: 1 04:03:00,707 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Number of entries to be appended = 1 04:03:00,707 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 04:03:00,707 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Append entry to log zero 04:03:00,707 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 1 is present for follower follower742 04:03:00,707 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Log size is now 1 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Sending AppendEntries to follower follower742: AppendEntries{term=6, leaderId=leader740, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:03:00,707 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower741, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:00,707 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached -1 (of expected -1) 04:03:00,707 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: awaiting more completions to resolve 1 synchronized callback(s) 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower741 updated: matchIndex: 0, nextIndex: 1 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:00,707 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 578.2 ms 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 1 is present for follower follower741 04:03:00,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Sending AppendEntries to follower follower741: AppendEntries{term=6, leaderId=leader740, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:03:00,711 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:00,711 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 4.482 ms 04:03:00,711 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 6.310 μs 04:03:00,711 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:00,711 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader740: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:00,711 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader740 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 04:03:00,711 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader740 (Leader): Setting last applied to 0 04:03:00,711 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader740: Moving last applied index from -1 to 0 04:03:00,711 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader740: Applying state for log index 0 data zero 04:03:00,711 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader740: applyState called: zero 04:03:00,713 PM [follower742-writer-136] [DEBUG] RaftStorageCompleter - follower742: 1 completion(s) pending 04:03:00,713 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 1 completion(s) pending 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower742: running 1 completion(s) 04:03:00,713 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: completed 1 commands in 6.272 ms 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower742: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:03:00,713 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: waiting to receive commands 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower742: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=6, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@1a8bcef8} 04:03:00,713 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: running 1 completion(s) 04:03:00,713 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 6.011 ms 04:03:00,713 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands 04:03:00,713 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:03:00,713 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower742: synchronized callbacks after 6.464 ms 04:03:00,713 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=6, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@71557373} 04:03:00,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader740, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:03:00,713 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: synchronized callbacks after 6.186 ms 04:03:00,713 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 2.363 ms 04:03:00,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader740: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): Number of entries to be appended = 1 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): Append entry to log one 04:03:00,713 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 6.080 μs 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower742: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): Log size is now 2 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower742, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:00,713 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower742: Lagging 0 entries behind leader leader740 and reached -1 (of expected -1) 04:03:00,713 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader740, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 04:03:00,714 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower742: awaiting more completions to resolve 1 synchronized callback(s) 04:03:00,714 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Number of entries to be appended = 1 04:03:00,714 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: received 1 commands after 262.9 μs 04:03:00,714 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 04:03:00,714 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Append entry to log one 04:03:00,714 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:00,714 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=1, applyTo=1] 04:03:00,714 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Log size is now 2 04:03:00,714 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:00,714 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower741, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:00,714 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 235.0 μs 04:03:00,714 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached -1 (of expected -1) 04:03:00,714 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:00,714 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: awaiting more completions to resolve 1 synchronized callback(s) 04:03:00,714 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 437.0 μs 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower742 updated: matchIndex: 1, nextIndex: 2 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader740 (Leader): Setting last applied to 1 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader740: Moving last applied index from 0 to 1 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader740: Applying state for log index 1 data one 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader740: applyState called: one 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader740: fakeSnapshot purging log to 0 for term 6 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 04:03:00,714 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 544.4 μs 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower741 updated: matchIndex: 1, nextIndex: 2 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 04:03:00,714 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:03:00,714 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:00,714 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:00,714 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 168.7 μs 04:03:00,714 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:00,715 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 1 completion(s) pending 04:03:00,715 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 801.2 μs 04:03:00,715 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands 04:03:00,715 PM [follower742-writer-136] [DEBUG] RaftStorageCompleter - follower742: 1 completion(s) pending 04:03:00,715 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: completed 1 commands in 1.090 ms 04:03:00,715 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: waiting to receive commands 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: running 1 completion(s) 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=6, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@43db92f5} 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: synchronized callbacks after 1.344 ms 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Commit index set to 1 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 1 (of expected -1) 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 1 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Setting last applied to 0 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower741: Moving last applied index from -1 to 0 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower741: Applying state for log index 0 data zero 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower741: applyState called: zero 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Setting last applied to 1 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower741: Moving last applied index from 0 to 1 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower741: Applying state for log index 1 data one 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower741: applyState called: one 04:03:00,715 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower741: fakeSnapshot purging log to 0 for term 6 04:03:00,715 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 885.9 μs [WARN] [08/11/2025 16:03:00.716] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower741] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower741, initialSyncDone=true] 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower742: running 1 completion(s) 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower742: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower742: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=6, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@6198a2c1} 04:03:00,716 PM [follower741-writer-135] [DEBUG] EntryJournalV1 - follower741: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower742: synchronized callbacks after 2.182 ms 04:03:00,716 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 1 completion(s) pending 04:03:00,716 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: running 1 completion(s) 04:03:00,716 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 218.2 μs 04:03:00,716 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): Commit index set to 1 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower742: Lagging 0 entries behind leader leader740 and reached 1 (of expected -1) 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower742 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 1 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower742 (Follower): Setting last applied to 0 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower742: Moving last applied index from -1 to 0 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower742: Applying state for log index 0 data zero 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower742: applyState called: zero 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower742 (Follower): Setting last applied to 1 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower742: Moving last applied index from 0 to 1 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower742: Applying state for log index 1 data one 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower742: applyState called: one 04:03:00,716 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower742: fakeSnapshot purging log to 0 for term 6 04:03:00,716 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: received 1 commands after 1.453 ms 04:03:00,716 PM [follower742-writer-136] [DEBUG] EntryJournalV1 - follower742: updated meta to JournalMeta[replayFrom=1, applyTo=2] [WARN] [08/11/2025 16:03:00.716] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower742] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower742, initialSyncDone=true] 04:03:00,716 PM [follower742-writer-136] [DEBUG] RaftStorageCompleter - follower742: 1 completion(s) pending 04:03:00,716 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: completed 1 commands in 162.4 μs 04:03:00,716 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: waiting to receive commands 04:03:00,716 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower742: running 1 completion(s) 04:03:00,914 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower742: Lagging 0 entries behind leader leader740 and reached 1 (of expected -1) 04:03:00,914 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 1 (of expected -1) 04:03:01,34 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 1 (of expected -1) 04:03:01,153 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 1 (of expected -1) 04:03:01,273 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 1 (of expected -1) 04:03:01,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader740: Persist data index=2 term=6 command=two 04:03:01,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader740: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:01,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Replicate message: identifier: MockIdentifier{value=two}, logIndex: 2, isSendImmediate: true 04:03:01,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: false, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 04:03:01,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 04:03:01,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 2 is present for follower follower741 04:03:01,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Sending AppendEntries to follower follower741: AppendEntries{term=6, leaderId=leader740, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 04:03:01,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader740: Persist data index=3 term=6 command=three 04:03:01,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader740: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:01,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Replicate message: identifier: MockIdentifier{value=three}, logIndex: 3, isSendImmediate: true 04:03:01,364 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader740, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 04:03:01,364 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 650.0 ms 04:03:01,365 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Number of entries to be appended = 1 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: false, followerNextIndex: 2, leaderLastIndex: 3, leaderSnapShotIndex: 0 04:03:01,365 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 04:03:01,365 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Append entry to log two 04:03:01,365 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 2, leaderLastIndex: 3, leaderSnapShotIndex: 0 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 2 is present for follower follower741 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader740: Persist data index=4 term=6 command=four 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader740: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:01,365 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Log size is now 2 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Replicate message: identifier: MockIdentifier{value=four}, logIndex: 4, isSendImmediate: true 04:03:01,365 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower741, logLastIndex=2, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: false, followerNextIndex: 2, leaderLastIndex: 4, leaderSnapShotIndex: 0 04:03:01,365 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 1 (of expected -1) 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 2, leaderLastIndex: 4, leaderSnapShotIndex: 0 04:03:01,365 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: awaiting more completions to resolve 1 synchronized callback(s) 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 2 is present for follower follower741 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower741 updated: matchIndex: 2, nextIndex: 3 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 3, leaderLastIndex: 4, leaderSnapShotIndex: 0 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 3 is present for follower follower741 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Sending AppendEntries to follower follower741: AppendEntries{term=6, leaderId=leader740, prevLogIndex=2, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}, JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 04:03:01,365 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 649.3 ms 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader740: Persist data index=5 term=6 command=five 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader740: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Replicate message: identifier: MockIdentifier{value=five}, logIndex: 5, isSendImmediate: true 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: false, followerNextIndex: 2, leaderLastIndex: 5, leaderSnapShotIndex: 0 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 3, leaderLastIndex: 5, leaderSnapShotIndex: 0 04:03:01,365 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 3 is present for follower follower741 04:03:01,365 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader740: Persist data index=6 term=6 command=six 04:03:01,365 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader740: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=true} 04:03:01,365 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader740 (Leader): Replicate message: identifier: MockIdentifier{value=six}, logIndex: 6, isSendImmediate: true 04:03:01,366 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: false, followerNextIndex: 2, leaderLastIndex: 6, leaderSnapShotIndex: 0 04:03:01,366 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 3, leaderLastIndex: 6, leaderSnapShotIndex: 0 04:03:01,366 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 3 is present for follower follower741 04:03:01,366 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,366 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 1 completion(s) pending 04:03:01,366 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 1.626 ms 04:03:01,366 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: running 1 completion(s) 04:03:01,366 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 4 commands after 4.370 μs 04:03:01,366 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 1.066 ms 04:03:01,366 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands 04:03:01,366 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=false} 04:03:01,366 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=6, command=two, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@106bb88e} 04:03:01,366 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,366 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: synchronized callbacks after 1.546 ms 04:03:01,366 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader740: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:01,366 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 04:03:01,366 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader740, prevLogIndex=2, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}, JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 04:03:01,367 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader740 (Leader): Setting last applied to 2 04:03:01,367 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Number of entries to be appended = 2 04:03:01,367 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader740: Moving last applied index from 1 to 2 04:03:01,367 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 04:03:01,367 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader740: Applying state for log index 2 data two 04:03:01,367 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Append entry to log three 04:03:01,367 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader740: applyState called: two 04:03:01,367 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:01,367 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Append entry to log four 04:03:01,367 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:01,367 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Log size is now 4 04:03:01,367 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower741, logLastIndex=4, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:01,367 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 1 (of expected -1) 04:03:01,367 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 2 commands after 405.6 μs 04:03:01,367 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader740: fakeSnapshot purging log to 1 for term 6 04:03:01,367 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: awaiting more completions to resolve 2 synchronized callback(s) 04:03:01,367 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower741 updated: matchIndex: 4, nextIndex: 5 04:03:01,367 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 5, leaderLastIndex: 6, leaderSnapShotIndex: 1 04:03:01,367 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 5 is present for follower follower741 04:03:01,367 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Sending AppendEntries to follower follower741: AppendEntries{term=6, leaderId=leader740, prevLogIndex=4, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}, JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 04:03:01,369 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 2 completion(s) pending 04:03:01,369 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 2 commands in 2.192 ms 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: running 2 completion(s) 04:03:01,369 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=false} 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=3, term=6, command=three, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@48c4b6aa} 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=false} 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=4, term=6, command=four, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@48c4b6aa} 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: synchronized callbacks after 2.309 ms 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader740, prevLogIndex=4, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}, JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Number of entries to be appended = 2 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): After cleanup, lastIndex: 4, entries to be added from: 0 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Append entry to log five 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Append entry to log six 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=true} 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Log size is now 6 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Commit index set to 2 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower741, logLastIndex=6, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:01,369 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 372.8 μs 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 2 (of expected -1) 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 6 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Setting last applied to 2 04:03:01,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower741: Moving last applied index from 1 to 2 04:03:01,369 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower741 updated: matchIndex: 6, nextIndex: 7 04:03:01,370 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower741: Applying state for log index 2 data two 04:03:01,370 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower741: applyState called: two 04:03:01,370 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 7, leaderLastIndex: 6, leaderSnapShotIndex: 1 04:03:01,370 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower741: fakeSnapshot purging log to 1 for term 6 04:03:01,370 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: awaiting more completions to resolve 2 synchronized callback(s) 04:03:01,371 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 1 completion(s) pending 04:03:01,371 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: running 1 completion(s) 04:03:01,371 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 1.252 ms 04:03:01,371 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=false} 04:03:01,371 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 2 commands after 3.670 μs 04:03:01,371 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=5, term=6, command=five, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@54848f26} 04:03:01,371 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: awaiting more completions to resolve 1 synchronized callback(s) 04:03:01,371 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 4 completion(s) pending 04:03:01,371 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 4 commands in 4.689 ms 04:03:01,371 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader740: running 4 completion(s) 04:03:01,371 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 3.070 μs 04:03:01,371 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader740: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:01,371 PM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - leader740: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=2, lastAppliedTerm=6, lastIndex=3, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=4, mandatoryTrim=false] 04:03:01,371 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader740: captured snapshot at lastSequenceNumber: 3 04:03:01,371 PM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - leader740: Persising snapshot at EntryInfo[index=2, term=6]/EntryInfo[index=3, term=6] 04:03:01,371 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader740: support() called 04:03:01,371 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=1, applyTo=3] 04:03:01,371 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,371 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 356.6 μs 04:03:01,371 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader740: log size 5 exceeds the snapshot batch count 4 - doing snapshotPreCommit with index 2 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - leader740: Removed in-memory snapshotted entries, adjusted snapshotIndex: 2 and term: 6 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader740 (Leader): Setting last applied to 3 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader740: Moving last applied index from 2 to 3 04:03:01,372 PM [leader740-%d851179] [DEBUG] RaftStorage - leader740: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-162D984F.tmp 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader740: Applying state for log index 3 data three 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader740: applyState called: three 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader740: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Applying to log - commitIndex: 4, lastAppliedIndex: 3 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader740 (Leader): Setting last applied to 4 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader740: Moving last applied index from 3 to 4 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader740: Applying state for log index 4 data four 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader740: applyState called: four 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader740: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Applying to log - commitIndex: 5, lastAppliedIndex: 4 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader740 (Leader): Setting last applied to 5 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader740: Moving last applied index from 4 to 5 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader740: Applying state for log index 5 data five 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader740: applyState called: five 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader740: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=true} 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): Applying to log - commitIndex: 6, lastAppliedIndex: 5 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader740 (Leader): Setting last applied to 6 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader740: Moving last applied index from 5 to 6 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader740: Applying state for log index 6 data six 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader740: applyState called: six 04:03:01,372 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,372 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 4 commands after 1.054 ms 04:03:01,373 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=1, applyTo=4] 04:03:01,373 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=1, applyTo=5] 04:03:01,373 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=1, applyTo=6] 04:03:01,373 PM [follower741-writer-135] [DEBUG] EntryJournalV1 - follower741: updated meta to JournalMeta[replayFrom=1, applyTo=3] 04:03:01,373 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=1, applyTo=7] 04:03:01,373 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 2 completion(s) pending 04:03:01,373 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 4 completion(s) pending 04:03:01,373 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: running 2 completion(s) 04:03:01,373 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=false} 04:03:01,373 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 4 commands in 396.4 μs 04:03:01,373 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader740: running 4 completion(s) 04:03:01,373 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:01,373 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=6, term=6, command=six, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@54848f26} 04:03:01,373 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: synchronized callbacks after 3.269 ms 04:03:01,373 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 2 commands in 2.188 ms 04:03:01,373 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands 04:03:01,375 PM [leader740-%d851179] [DEBUG] RaftStorage - leader740: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-162D984F.v1 04:03:01,375 PM [leader740-%d851179] [DEBUG] RaftStorage - leader740: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-162D984F.v1 04:03:01,375 PM [leader740-%d851179] [DEBUG] RaftStorage - leader740: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1434-0899B446.v1 04:03:01,375 PM [leader740-%d851179] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/TermInfo.properties 04:03:01,375 PM [leader740-%d851179] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/journal-v1-1.log 04:03:01,375 PM [leader740-%d851179] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/metadata-v1-1.log 04:03:01,375 PM [leader740-%d851179] [DEBUG] RaftStorage - leader740: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:00.144290886Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1434-0899B446.v1, size=221, entryCompress=NONE, stateCompress=NONE} 04:03:01,375 PM [leader740-%d851179] [DEBUG] RaftStorage - leader740: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:01.372086863Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-162D984F.v1, size=374, entryCompress=NONE, stateCompress=NONE} 04:03:01,375 PM [leader740-%d851179] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,375 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,375 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader740: snapshot is durable as of 2025-08-11T16:03:01.372086863Z 04:03:01,375 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader740: Snapshot success, discarding journal entries up to 3 04:03:01,375 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorSnapshotMessageSupport - leader740: SnapshotComplete received 04:03:01,375 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 2.546 ms 04:03:01,376 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=4, applyTo=7] 04:03:01,376 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: discarded entries up to (and including) 3 04:03:01,376 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,376 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 187.8 μs 04:03:01,376 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,376 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:01,465 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-162D984F.v1 04:03:01,465 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/TermInfo.properties 04:03:01,465 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/journal-v1-1.log 04:03:01,465 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/metadata-v1-1.log 04:03:01,465 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: picked SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:01.372086863Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-162D984F.v1, size=374, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:01,465 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaInstallSnapshot: sending 1 more payload to trigger second snapshot 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader740: Persist data index=7 term=6 command=seven 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader740: Append log entry and persist JournaledLogEntry{index=7, term=6, command=seven, pending=true} 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): Replicate message: identifier: MockIdentifier{value=seven}, logIndex: 7, isSendImmediate: true 04:03:01,466 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 90.13 ms 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: false, followerNextIndex: 2, leaderLastIndex: 7, leaderSnapShotIndex: 2 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 7, leaderLastIndex: 7, leaderSnapShotIndex: 2 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 7 is present for follower follower741 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): Sending AppendEntries to follower follower741: AppendEntries{term=6, leaderId=leader740, prevLogIndex=6, prevLogTerm=6, leaderCommit=6, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=7, term=6, command=seven, pending=true}]} 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader740, prevLogIndex=6, prevLogTerm=6, leaderCommit=6, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=7, term=6, command=seven, pending=true}]} 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Number of entries to be appended = 1 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): After cleanup, lastIndex: 6, entries to be added from: 0 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Append entry to log seven 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: Append log entry and persist JournaledLogEntry{index=7, term=6, command=seven, pending=true} 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Log size is now 6 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Commit index set to 6 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower741, logLastIndex=7, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 6 (of expected -1) 04:03:01,466 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 93.41 ms 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 6, context.getLastApplied(): 2, lastIndex(): 7 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Setting last applied to 3 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower741: Moving last applied index from 2 to 3 04:03:01,466 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower741: Applying state for log index 3 data three 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower741: applyState called: three 04:03:01,467 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower741 updated: matchIndex: 7, nextIndex: 8 04:03:01,467 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 8, leaderLastIndex: 7, leaderSnapShotIndex: 2 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Setting last applied to 4 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower741: Moving last applied index from 3 to 4 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower741: Applying state for log index 4 data four 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower741: applyState called: four 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Setting last applied to 5 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower741: Moving last applied index from 4 to 5 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower741: Applying state for log index 5 data five 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower741: applyState called: five 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Setting last applied to 6 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower741: Moving last applied index from 5 to 6 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower741: Applying state for log index 6 data six 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower741: applyState called: six 04:03:01,467 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: awaiting more completions to resolve 1 synchronized callback(s) 04:03:01,467 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,467 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,467 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 1.151 ms 04:03:01,467 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:01,467 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader740: persist complete JournaledLogEntry{index=7, term=6, command=seven, pending=true} 04:03:01,467 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader740: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=6, lastAppliedTerm=6, lastIndex=7, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=1, mandatoryTrim=false] 04:03:01,467 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader740: captured snapshot at lastSequenceNumber: 7 04:03:01,467 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader740: Persising snapshot at EntryInfo[index=6, term=6]/EntryInfo[index=7, term=6] 04:03:01,467 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader740: support() called 04:03:01,468 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 1 completion(s) pending 04:03:01,468 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader740: log size 5 exceeds the snapshot batch count 4 - doing snapshotPreCommit with index 6 04:03:01,468 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: running 1 completion(s) 04:03:01,468 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 1.403 ms 04:03:01,468 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader740: Removed in-memory snapshotted entries, adjusted snapshotIndex: 6 and term: 6 04:03:01,468 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: persist complete JournaledLogEntry{index=7, term=6, command=seven, pending=false} 04:03:01,468 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 4.050 μs 04:03:01,468 PM [leader740-%d851180] [DEBUG] RaftStorage - leader740: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-1BE8E65D.tmp 04:03:01,468 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=7, term=6, command=seven, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@176ac8b2} 04:03:01,468 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader740 (Leader): Applying to log - commitIndex: 7, lastAppliedIndex: 6 04:03:01,468 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader740 (Leader): Setting last applied to 7 04:03:01,468 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: synchronized callbacks after 997.8 μs 04:03:01,468 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader740: Moving last applied index from 6 to 7 04:03:01,468 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader740: Applying state for log index 7 data seven 04:03:01,468 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader740: applyState called: seven 04:03:01,468 PM [follower741-writer-135] [DEBUG] EntryJournalV1 - follower741: updated meta to JournalMeta[replayFrom=1, applyTo=7] 04:03:01,468 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 1 completion(s) pending 04:03:01,468 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 257.1 μs 04:03:01,468 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 1.113 ms 04:03:01,468 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands 04:03:01,468 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower741: running 1 completion(s) 04:03:01,468 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=4, applyTo=8] 04:03:01,468 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,468 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 177.9 μs 04:03:01,468 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:01,468 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,470 PM [leader740-%d851180] [DEBUG] RaftStorage - leader740: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-1BE8E65D.v1 04:03:01,470 PM [leader740-%d851180] [DEBUG] RaftStorage - leader740: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-162D984F.v1 04:03:01,470 PM [leader740-%d851180] [DEBUG] RaftStorage - leader740: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-1BE8E65D.v1 04:03:01,470 PM [leader740-%d851180] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/TermInfo.properties 04:03:01,470 PM [leader740-%d851180] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/journal-v1-1.log 04:03:01,470 PM [leader740-%d851180] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/metadata-v1-1.log 04:03:01,470 PM [leader740-%d851180] [DEBUG] RaftStorage - leader740: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:01.372086863Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-162D984F.v1, size=374, entryCompress=NONE, stateCompress=NONE} 04:03:01,470 PM [leader740-%d851180] [DEBUG] RaftStorage - leader740: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-08-11T16:03:01.468248157Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-1BE8E65D.v1, size=442, entryCompress=NONE, stateCompress=NONE} 04:03:01,470 PM [leader740-%d851180] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,471 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,471 PM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - leader740: snapshot is durable as of 2025-08-11T16:03:01.468248157Z 04:03:01,471 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader740: Snapshot success, discarding journal entries up to 7 04:03:01,471 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorSnapshotMessageSupport - leader740: SnapshotComplete received 04:03:01,471 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 2.369 ms 04:03:01,471 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=8, applyTo=8] 04:03:01,471 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: discarded entries up to (and including) 7 04:03:01,471 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,471 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,471 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 181.7 μs 04:03:01,471 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:01,633 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower741 (Follower): Commit index set to 7 04:03:01,633 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 7 (of expected -1) 04:03:01,633 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower741 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 7, context.getLastApplied(): 6, lastIndex(): 7 04:03:01,633 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower741 (Follower): Setting last applied to 7 04:03:01,633 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower741: Moving last applied index from 6 to 7 04:03:01,633 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower741: Applying state for log index 7 data seven 04:03:01,633 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower741: applyState called: seven 04:03:01,634 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 165.3 ms 04:03:01,634 PM [follower741-writer-135] [DEBUG] EntryJournalV1 - follower741: updated meta to JournalMeta[replayFrom=1, applyTo=8] 04:03:01,634 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 1 completion(s) pending 04:03:01,634 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 276.5 μs 04:03:01,634 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands 04:03:01,634 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower741: running 1 completion(s) 04:03:01,667 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-1BE8E65D.v1 04:03:01,667 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/TermInfo.properties 04:03:01,667 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/journal-v1-1.log 04:03:01,667 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/metadata-v1-1.log 04:03:01,668 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: picked SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-08-11T16:03:01.468248157Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-1BE8E65D.v1, size=442, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:01,669 PM [main] [DEBUG] PeerInfos - leader740: Updated server config: isVoting: true, peers: [PeerInfo [id=follower742, address=pekko://test/user/follower742, votingState=NON_VOTING], PeerInfo [id=follower741, address=pekko://test/user/follower741, votingState=NON_VOTING]] 04:03:01,669 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader740: Persist data index=8 term=6 command=VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]} 04:03:01,669 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader740: Append log entry and persist JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]}, pending=true} 04:03:01,669 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Replicate message: identifier: MockIdentifier{value=serverConfig}, logIndex: 8, isSendImmediate: true 04:03:01,669 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 198.2 ms 04:03:01,669 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower742: active: false, followerNextIndex: 2, leaderLastIndex: 8, leaderSnapShotIndex: 6 04:03:01,669 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 8, leaderLastIndex: 8, leaderSnapShotIndex: 6 04:03:01,669 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): sendAppendEntries: 8 is present for follower follower741 04:03:01,670 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader740 (Leader): Sending AppendEntries to follower follower741: AppendEntries{term=6, leaderId=leader740, prevLogIndex=7, prevLogTerm=6, leaderCommit=7, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]}, pending=true}]} 04:03:01,670 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader740, prevLogIndex=7, prevLogTerm=6, leaderCommit=7, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]}, pending=true}]} 04:03:01,670 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Number of entries to be appended = 1 04:03:01,670 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): After cleanup, lastIndex: 7, entries to be added from: 0 04:03:01,670 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]} 04:03:01,670 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: Append log entry and persist JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]}, pending=true} 04:03:01,670 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PeerInfos - follower741: Updated server config: isVoting: false, peers: [PeerInfo [id=leader740, address=pekko://test/user/leader740, votingState=VOTING], PeerInfo [id=follower742, address=pekko://test/user/follower742, votingState=NON_VOTING]] 04:03:01,670 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Log size is now 7 04:03:01,670 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower741, logLastIndex=8, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:01,670 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 35.90 ms 04:03:01,670 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 7 (of expected -1) 04:03:01,670 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: awaiting more completions to resolve 1 synchronized callback(s) 04:03:01,670 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,670 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 1.068 ms 04:03:01,670 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:01,671 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 1 completion(s) pending 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: running 1 completion(s) 04:03:01,671 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 975.0 μs 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower741: persist complete JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]}, pending=false} 04:03:01,671 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@1f11a693} 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: synchronized callbacks after 1.025 ms 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader740: persist complete JournaledLogEntry{index=8, term=6, command=VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]}, pending=true} 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): Applying to log - commitIndex: 8, lastAppliedIndex: 7 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader740 (Leader): Setting last applied to 8 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader740: Moving last applied index from 7 to 8 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader740: Applying state for log index 8 data VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]} 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower741 updated: matchIndex: 8, nextIndex: 9 04:03:01,671 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 730.6 μs 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): Checking sendAppendEntries for follower follower741: active: true, followerNextIndex: 9, leaderLastIndex: 8, leaderSnapShotIndex: 6 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): Commit index set to 8 04:03:01,671 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=8, applyTo=9] 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 8 (of expected -1) 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower741 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 8, context.getLastApplied(): 7, lastIndex(): 8 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower741 (Follower): Setting last applied to 8 04:03:01,671 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower741: Moving last applied index from 7 to 8 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower741: Applying state for log index 8 data VotingConfig{serverInfo=[ServerInfo[peerId=leader740, isVoting=true], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=follower742, isVoting=false]]} 04:03:01,671 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 146.8 μs 04:03:01,671 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:01,671 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,671 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 454.4 μs 04:03:01,671 PM [follower741-writer-135] [DEBUG] EntryJournalV1 - follower741: updated meta to JournalMeta[replayFrom=1, applyTo=9] 04:03:01,671 PM [follower741-writer-135] [DEBUG] RaftStorageCompleter - follower741: 1 completion(s) pending 04:03:01,672 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower741: running 1 completion(s) 04:03:01,672 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 109.8 μs 04:03:01,672 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: waiting to receive commands 04:03:01,720 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - verifyInstallSnapshotToLaggingFollower starting 04:03:01,753 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower742: Lagging -2 entries behind leader leader740 and reached 1 (of expected -1) 04:03:01,754 PM [test-pekko.actor.default-dispatcher-12] [WARN] AbstractLeader - leader740 (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply{term=6, success=true, followerId=follower742, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, timeSinceLastActivity : 839, lastApplied : 8, commitIndex : 8 04:03:01,754 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractLeader - leader740 (Leader): Initiating install snapshot to follower follower742: follower nextIndex: 2, leader snapshotIndex: 6, leader lastIndex: 8, leader log size: 2 04:03:01,754 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader740: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=8, lastAppliedTerm=6, lastIndex=8, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower742 04:03:01,754 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader740: last applied journal index prior to capture: 9 04:03:01,754 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader740: support() called 04:03:01,754 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower742: Lagging -2 entries behind leader leader740 and reached 1 (of expected -1) 04:03:01,755 PM [leader740-%d851181] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,756 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,756 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader740: Persising snapshot at EntryInfo[index=8, term=6]/EntryInfo[index=8, term=6] 04:03:01,756 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader740: support() called 04:03:01,757 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader740: Removed in-memory snapshotted entries, adjusted snapshotIndex: 6 and term: 6 04:03:01,757 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): sendInstallSnapshot 04:03:01,757 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] LeaderInstallSnapshotState - leader740 (Leader): Snapshot 396 bytes, total chunks to send: 1 04:03:01,757 PM [leader740-%d851182] [DEBUG] RaftStorage - leader740: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-2D1FD7DF.tmp 04:03:01,757 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] LeaderInstallSnapshotState - leader740 (Leader): Next chunk: total length=396, offset=0, size=396, hashCode=-153134944 04:03:01,757 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): next snapshot chunk size for follower follower742: 396 04:03:01,757 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower742 (Follower): handleInstallSnapshot: InstallSnapshot{term=6, leaderId=leader740, lastIncludedIndex=8, lastIncludedTerm=6, datasize=396, chunk=1/1, lastChunkHashCode=OptionalInt[-1], votingConfig=VotingConfig{serverInfo=[ServerInfo[peerId=follower742, isVoting=false], ServerInfo[peerId=follower741, isVoting=false], ServerInfo[peerId=leader740, isVoting=true]]}} 04:03:01,757 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower742: Lagging 7 entries behind leader leader740 and reached 1 (of expected -1) 04:03:01,757 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotTracker - follower742 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 04:03:01,757 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower742 (Follower): Snapshot received from leader: leader740 04:03:01,757 PM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - follower742: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=396}} 04:03:01,757 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower742: support() called 04:03:01,758 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader740 (Leader): InstallSnapshot sent to follower Vector(user, follower742), Chunk: 1/1 04:03:01,758 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower742: Converted InstallSnapshot from leader: leader740 to state 04:03:01,759 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower742: last applied journal index prior to persisting applied snapshot: 2 04:03:01,759 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower742: support() called 04:03:01,759 PM [follower742-%d871040] [DEBUG] RaftStorage - follower742: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower742/snapshot-00000000689A1435-2D42F69C.tmp 04:03:01,759 PM [leader740-%d851182] [DEBUG] RaftStorage - leader740: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-2D1FD7DF.v1 04:03:01,760 PM [leader740-%d851182] [DEBUG] RaftStorage - leader740: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-2D1FD7DF.v1 04:03:01,760 PM [leader740-%d851182] [DEBUG] RaftStorage - leader740: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-1BE8E65D.v1 04:03:01,760 PM [leader740-%d851182] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/TermInfo.properties 04:03:01,760 PM [leader740-%d851182] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/journal-v1-1.log 04:03:01,760 PM [leader740-%d851182] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/metadata-v1-1.log 04:03:01,760 PM [leader740-%d851182] [DEBUG] RaftStorage - leader740: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-08-11T16:03:01.468248157Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-1BE8E65D.v1, size=442, entryCompress=NONE, stateCompress=NONE} 04:03:01,760 PM [leader740-%d851182] [DEBUG] RaftStorage - leader740: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=8, term=6], timestamp=2025-08-11T16:03:01.757061599Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-2D1FD7DF.v1, size=500, entryCompress=NONE, stateCompress=NONE} 04:03:01,760 PM [leader740-%d851182] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,760 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,760 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader740: snapshot is durable as of 2025-08-11T16:03:01.757061599Z 04:03:01,760 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - leader740: Snapshot success, discarding journal entries up to 9 04:03:01,760 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 89.03 ms 04:03:01,761 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: updated meta to JournalMeta[replayFrom=10, applyTo=9] 04:03:01,761 PM [leader740-writer-137] [DEBUG] EntryJournalV1 - leader740: discarded entries up to (and including) 9 04:03:01,761 PM [leader740-writer-137] [DEBUG] RaftStorageCompleter - leader740: 1 completion(s) pending 04:03:01,761 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 148.7 μs 04:03:01,761 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: waiting to receive commands 04:03:01,761 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader740: running 1 completion(s) 04:03:01,761 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorSnapshotMessageSupport - leader740: SnapshotComplete received 04:03:01,761 PM [follower742-%d871040] [DEBUG] RaftStorage - follower742: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower742/snapshot-00000000689A1435-2D42F69C.v1 04:03:01,761 PM [follower742-%d871040] [DEBUG] RaftStorage - follower742: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower742/snapshot-00000000689A1435-2D42F69C.v1 04:03:01,762 PM [follower742-%d871040] [DEBUG] RaftStorage - follower742: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower742/TermInfo.properties 04:03:01,762 PM [follower742-%d871040] [DEBUG] RaftStorage - follower742: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower742/journal-v1-1.log 04:03:01,762 PM [follower742-%d871040] [DEBUG] RaftStorage - follower742: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower742/metadata-v1-1.log 04:03:01,762 PM [follower742-%d871040] [DEBUG] RaftStorage - follower742: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=8, term=6], timestamp=2025-08-11T16:03:01.759363228Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/follower742/snapshot-00000000689A1435-2D42F69C.v1, size=500, entryCompress=NONE, stateCompress=NONE} 04:03:01,762 PM [follower742-%d871040] [DEBUG] RaftStorageCompleter - follower742: 1 completion(s) pending 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower742: running 1 completion(s) 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - follower742: snapshot is durable as of 2025-08-11T16:03:01.759363228Z 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - follower742: Updated server config: isVoting: false, peers: [PeerInfo [id=leader740, address=pekko://test/user/leader740, votingState=VOTING], PeerInfo [id=follower741, address=pekko://test/user/follower741, votingState=NON_VOTING]] 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower742: support() called 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower742: applySnapshot called 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower742 (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=6, followerId=follower742, chunkIndex=1, success=true} 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower742: Snapshot success, discarding journal entries up to 2 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorSnapshotMessageSupport - follower742: SnapshotComplete received 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader740 (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=6, followerId=follower742, chunkIndex=1, success=true} 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractLeader - leader740 (Leader): Snapshot successfully installed on follower follower742 (last chunk 1) - matchIndex set to 8, nextIndex set to 9 04:03:01,762 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: received 1 commands after 1.046 s 04:03:01,762 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader740: fakeSnapshot purging log to 7 for term 6 04:03:01,762 PM [follower742-writer-136] [DEBUG] EntryJournalV1 - follower742: updated meta to JournalMeta[replayFrom=3, applyTo=2] 04:03:01,763 PM [follower742-writer-136] [DEBUG] EntryJournalV1 - follower742: discarded entries up to (and including) 2 04:03:01,763 PM [follower742-writer-136] [DEBUG] RaftStorageCompleter - follower742: 1 completion(s) pending 04:03:01,763 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower742: running 1 completion(s) 04:03:01,763 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: completed 1 commands in 201.2 μs 04:03:01,763 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: waiting to receive commands 04:03:01,874 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower742: Lagging 0 entries behind leader leader740 and reached 8 (of expected -1) 04:03:01,874 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 8 (of expected -1) 04:03:01,874 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower741: fakeSnapshot purging log to 7 for term 6 04:03:01,974 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-2D1FD7DF.v1 04:03:01,974 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/TermInfo.properties 04:03:01,974 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/journal-v1-1.log 04:03:01,974 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/metadata-v1-1.log 04:03:01,974 PM [awaitility-thread] [DEBUG] RaftStorage - leader740: picked SnapshotFileV1{lastIncluded=EntryInfo[index=8, term=6], timestamp=2025-08-11T16:03:01.757061599Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest18059119242905948007/leader740/snapshot-00000000689A1435-2D1FD7DF.v1, size=500, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:01,993 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower741: Lagging 0 entries behind leader leader740 and reached 8 (of expected -1) 04:03:01,993 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower742: Lagging 0 entries behind leader leader740 and reached 8 (of expected -1) 04:03:02,26 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - verifyInstallSnapshotToLaggingFollower complete 04:03:02,26 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaInstallSnapshot complete 04:03:02,27 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower741-collector743#1824571075] 04:03:02,38 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower741] 04:03:02,38 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EnabledRaftStorage - follower741: terminating thread VirtualThread[#907,follower741-writer-135]/waiting 04:03:02,38 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: received 1 commands after 366.4 ms 04:03:02,38 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: completed 1 commands in 81.19 μs 04:03:02,38 PM [follower741-writer-135] [DEBUG] JournalWriteTask - follower741: journal writer stopped 04:03:02,38 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower741: journal closed 04:03:02,38 PM [test-pekko.actor.default-dispatcher-15] [INFO] EnabledRaftStorage - follower741: journal closed 04:03:02,38 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower741: stopped executor with 0 remaining tasks 04:03:02,38 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower741: stopped executor with 0 remaining tasks 04:03:02,39 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower742-collector744#1906113902] 04:03:02,39 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower742] 04:03:02,39 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EnabledRaftStorage - follower742: terminating thread VirtualThread[#908,follower742-writer-136]/waiting 04:03:02,39 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: received 1 commands after 276.4 ms 04:03:02,39 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: completed 1 commands in 10.15 μs 04:03:02,39 PM [follower742-writer-136] [DEBUG] JournalWriteTask - follower742: journal writer stopped 04:03:02,39 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower742: journal closed 04:03:02,39 PM [test-pekko.actor.default-dispatcher-15] [INFO] EnabledRaftStorage - follower742: journal closed 04:03:02,39 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower742: stopped executor with 0 remaining tasks 04:03:02,39 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower742: stopped executor with 0 remaining tasks 04:03:02,39 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader740-collector745#653481273] 04:03:02,40 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader740] 04:03:02,40 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EnabledRaftStorage - leader740: terminating thread VirtualThread[#909,leader740-writer-137]/waiting 04:03:02,40 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: received 1 commands after 279.1 ms 04:03:02,40 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: completed 1 commands in 9.380 μs 04:03:02,40 PM [leader740-writer-137] [DEBUG] JournalWriteTask - leader740: journal writer stopped 04:03:02,40 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader740: journal closed 04:03:02,40 PM [test-pekko.actor.default-dispatcher-15] [INFO] EnabledRaftStorage - leader740: journal closed 04:03:02,40 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader740: stopped executor with 0 remaining tasks 04:03:02,40 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader740: stopped executor with 0 remaining tasks 04:03:02,42 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaAppendEntries starting [WARN] [08/11/2025 16:03:02.043] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower750]: org.opendaylight.controller.cluster.common.actor.Monitor@62e947e1 04:03:02,44 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower750: Starting recovery with journal batch size 1000 04:03:02,45 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower750: started executor 04:03:02,45 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower750: started executor 04:03:02,45 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower750: metadata journal open: replayFrom=1 applyTo=0 04:03:02,45 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower750: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:02,45 PM [test-pekko.actor.default-dispatcher-15] [INFO] EnabledRaftStorage - follower750: journal open: applyTo=0 04:03:02,46 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: journal writer started 04:03:02,46 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:02,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/follower750/TermInfo.properties does not exist 04:03:02,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower750: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/follower750/journal-v1-1.log 04:03:02,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower750: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/follower750/metadata-v1-1.log 04:03:02,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower750: no eligible files found 04:03:02,47 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower750: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/11/2025 16:03:02.047] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower751]: org.opendaylight.controller.cluster.common.actor.Monitor@59c628e4 04:03:02,47 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower751: Starting recovery with journal batch size 1000 04:03:02,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower751: started executor 04:03:02,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower751: started executor 04:03:02,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower751: metadata journal open: replayFrom=1 applyTo=0 04:03:02,49 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower751: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:02,49 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - follower751: journal open: applyTo=0 04:03:02,49 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: journal writer started 04:03:02,49 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: waiting to receive commands 04:03:02,49 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/follower751/TermInfo.properties does not exist 04:03:02,49 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower750: Set currentTerm=0, votedFor=null 04:03:02,49 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower750: Local TermInfo store seeded with TermInfo{term=0} 04:03:02,49 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower750: Pekko recovery completed and can restore from snapshot 04:03:02,49 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower751: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/follower751/journal-v1-1.log 04:03:02,49 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower750: starting journal recovery 04:03:02,49 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower751: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/follower751/metadata-v1-1.log 04:03:02,50 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower750: Moving last applied index from -1 to -1 04:03:02,50 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower751: no eligible files found 04:03:02,50 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower750: empty journal: appending 0 entries 04:03:02,50 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower751: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:02,50 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower750: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:02,50 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower750: journal recovery completedin 201.4 μs with journalIndex=1 04:03:02,50 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower750: Journal recovery completed and can restore from snapshot [WARN] [08/11/2025 16:03:02.050] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader749]: org.opendaylight.controller.cluster.common.actor.Monitor@47cc2e2c 04:03:02,51 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - leader749: Starting recovery with journal batch size 1000 04:03:02,51 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader749: started executor 04:03:02,51 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader749: started executor 04:03:02,52 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader749: metadata journal open: replayFrom=1 applyTo=0 04:03:02,52 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader749: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:02,52 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - leader749: journal open: applyTo=0 04:03:02,52 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: journal writer started 04:03:02,52 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: waiting to receive commands 04:03:02,53 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/TermInfo.properties does not exist 04:03:02,53 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/journal-v1-1.log 04:03:02,53 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/metadata-v1-1.log 04:03:02,53 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: no eligible files found 04:03:02,53 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - leader749: Set currentTerm=5, votedFor=leader749 04:03:02,53 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - leader749: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:02,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower751: Set currentTerm=0, votedFor=null 04:03:02,53 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower751: Local TermInfo store seeded with TermInfo{term=0} 04:03:02,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower751: Pekko recovery completed and can restore from snapshot 04:03:02,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower751: starting journal recovery 04:03:02,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower751: Moving last applied index from -1 to -1 04:03:02,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower751: empty journal: appending 0 entries 04:03:02,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower751: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:02,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower751: journal recovery completedin 559.4 μs with journalIndex=1 04:03:02,54 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower751: Journal recovery completed and can restore from snapshot 04:03:02,55 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - leader749: Set currentTerm=5, votedFor=leader749 04:03:02,55 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - leader749: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader749} 04:03:02,55 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - leader749: taking snapshot to clear Pekko persistence to 1 04:03:02,55 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Recovery - leader749: no applied entries in recovery log, re-snapshotting EntryInfo[index=-1, term=-1] 04:03:02,55 PM [test-pekko.actor.default-dispatcher-7] [INFO] Recovery - leader749: Taking snapshot on entry with index -1 04:03:02,55 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader749: support() called 04:03:02,56 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1436-03564B92.tmp 04:03:02,58 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1436-03564B92.v1 04:03:02,58 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/TermInfo.properties 04:03:02,58 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/journal-v1-1.log 04:03:02,58 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/metadata-v1-1.log 04:03:02,58 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1436-03564B92.v1 04:03:02,58 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:02.055987090Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1436-03564B92.v1, size=221, entryCompress=NONE, stateCompress=NONE} 04:03:02,58 PM [test-pekko.actor.default-dispatcher-7] [INFO] Recovery - leader749: Snapshot completed in 3.210 ms, resetting timer for the next recovery snapshot 04:03:02,58 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader749: Pekko recovery completed and cannot restore from snapshot 04:03:02,59 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader749: starting journal recovery 04:03:02,59 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader749: Moving last applied index from -1 to -1 04:03:02,59 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - leader749: empty journal: appending 0 entries 04:03:02,59 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - leader749: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:02,59 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - leader749: journal recovery completedin 143.8 μs with journalIndex=1 04:03:02,59 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader749: Journal recovery completed and cannot restore from snapshot 04:03:02,59 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader749: ignoring DeleteMessagesSuccess(1) 04:03:02,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 04:03:02,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader749 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.10:17355, Removed) 04:03:02,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader749 (Follower): Received ElectionTimeout - switching to Candidate 04:03:02,533 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader749 (Candidate): Election: Candidate has following voting peers: [follower751, follower750] 04:03:02,537 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader749: Set currentTerm=6, votedFor=leader749 04:03:02,537 PM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader749 (Candidate): Starting new election term 6 04:03:02,538 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader749 (Candidate): Sending RequestVote{term=6, candidateId=leader749, lastLogIndex=-1, lastLogTerm=-1} to peer follower751 04:03:02,538 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader749 (Candidate): Sending RequestVote{term=6, candidateId=leader749, lastLogIndex=-1, lastLogTerm=-1} to peer follower750 04:03:02,538 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader749 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 04:03:02,538 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower751 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 04:03:02,538 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower751 (Follower): Cluster state: [] 04:03:02,538 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower751 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader749, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 04:03:02,538 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower750 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 04:03:02,538 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower750 (Follower): Cluster state: [] 04:03:02,538 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower751 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader749, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 04:03:02,538 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower750 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader749, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 04:03:02,538 PM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower750 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader749, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 04:03:02,542 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower750: Set currentTerm=6, votedFor=null 04:03:02,542 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower750 (Follower): In requestVote: RequestVote{term=6, candidateId=leader749, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 04:03:02,543 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower751: Set currentTerm=6, votedFor=null 04:03:02,543 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower751 (Follower): In requestVote: RequestVote{term=6, candidateId=leader749, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 04:03:02,545 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower750: Set currentTerm=6, votedFor=leader749 04:03:02,545 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower750 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 04:03:02,545 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader749 (Candidate): RaftRPC message received RequestVoteReply{term=6, voteGranted=true}, my term is 6 04:03:02,545 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader749 (Candidate): handleRequestVoteReply: RequestVoteReply{term=6, voteGranted=true}, current voteCount: 1 04:03:02,545 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Election: Leader has following peers: [follower751, follower750] 04:03:02,546 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:02,546 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:02,546 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader749 (Candidate) :- Switching from behavior Candidate to Leader, election term: 6 04:03:02,546 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower751: Set currentTerm=6, votedFor=leader749 04:03:02,546 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower751 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 04:03:02,546 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower750: Last sync leader does not match current leader leader749, need to catch up to -1 04:03:02,546 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower750 updated: matchIndex: -1, nextIndex: 0 04:03:02,546 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower751: Last sync leader does not match current leader leader749, need to catch up to -1 04:03:02,546 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:02,546 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower751 updated: matchIndex: -1, nextIndex: 0 [WARN] [08/11/2025 16:03:02.546] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower750] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower750, initialSyncDone=false] 04:03:02,546 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/11/2025 16:03:02.546] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower751] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower751, initialSyncDone=false] 04:03:02,562 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - Leader created and elected 04:03:02,663 PM [awaitility-thread] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/TermInfo.properties 04:03:02,663 PM [awaitility-thread] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/journal-v1-1.log 04:03:02,663 PM [awaitility-thread] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/metadata-v1-1.log 04:03:02,663 PM [awaitility-thread] [DEBUG] RaftStorage - leader749: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1436-03564B92.v1 04:03:02,663 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower751: Lagging 0 entries behind leader leader749 and reached -1 (of expected -1) 04:03:02,663 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached -1 (of expected -1) [WARN] [08/11/2025 16:03:02.664] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower751] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower751, initialSyncDone=true] [WARN] [08/11/2025 16:03:02.664] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower750] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower750, initialSyncDone=true] 04:03:02,664 PM [awaitility-thread] [DEBUG] RaftStorage - leader749: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:02.055987090Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1436-03564B92.v1, size=221, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:02,664 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader749: Persist data index=0 term=6 command=zero 04:03:02,664 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader749: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:02,665 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 612.4 ms 04:03:02,665 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 04:03:02,665 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:02,665 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 0 is present for follower follower751 04:03:02,665 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower751: AppendEntries{term=6, leaderId=leader749, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:02,665 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:02,665 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 0 is present for follower follower750 04:03:02,665 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader749, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:02,665 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): Number of entries to be appended = 1 04:03:02,666 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 04:03:02,665 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower750: AppendEntries{term=6, leaderId=leader749, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:02,666 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): Append entry to log zero 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader749: Persist data index=1 term=6 command=one 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader749: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:03:02,666 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader749, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:02,666 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): Number of entries to be appended = 1 04:03:02,666 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 04:03:02,666 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): Append entry to log zero 04:03:02,666 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower750: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 0 is present for follower follower751 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 0 is present for follower follower750 04:03:02,666 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): Log size is now 1 04:03:02,666 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower750, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:02,666 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached -1 (of expected -1) 04:03:02,666 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower750: awaiting more completions to resolve 1 synchronized callback(s) 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower750 updated: matchIndex: 0, nextIndex: 1 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 1 is present for follower follower750 04:03:02,666 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 620.5 ms 04:03:02,666 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower750: AppendEntries{term=6, leaderId=leader749, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:03:02,666 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower751: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:02,666 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): Log size is now 1 04:03:02,666 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower751, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:02,666 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower751: Lagging 0 entries behind leader leader749 and reached -1 (of expected -1) 04:03:02,667 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower751: awaiting more completions to resolve 1 synchronized callback(s) 04:03:02,666 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: received 1 commands after 617.6 ms 04:03:02,667 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower751 updated: matchIndex: 0, nextIndex: 1 04:03:02,667 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:02,667 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 1 is present for follower follower751 04:03:02,667 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower751: AppendEntries{term=6, leaderId=leader749, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:03:02,670 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:02,670 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 5.645 ms 04:03:02,670 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 4.190 μs 04:03:02,670 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:02,670 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader749: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:02,671 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 04:03:02,671 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - leader749 (Leader): Setting last applied to 0 04:03:02,671 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader749: Moving last applied index from -1 to 0 04:03:02,671 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader749: Applying state for log index 0 data zero 04:03:02,671 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader749: applyState called: zero 04:03:02,672 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:02,672 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 1.436 ms 04:03:02,672 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:02,672 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 3.460 μs 04:03:02,672 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader749: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:03:02,672 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:02,672 PM [follower751-writer-139] [DEBUG] RaftStorageCompleter - follower751: 1 completion(s) pending 04:03:02,672 PM [leader749-writer-140] [DEBUG] EntryJournalV1 - leader749: updated meta to JournalMeta[replayFrom=1, applyTo=1] 04:03:02,672 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 5.977 ms 04:03:02,672 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:02,672 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:02,672 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: completed 1 commands in 5.542 ms 04:03:02,672 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:02,672 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 245.2 μs 04:03:02,672 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: waiting to receive commands 04:03:02,672 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: waiting to receive commands 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower750: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:03:02,672 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower751: running 1 completion(s) 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower750: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=6, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@231abe73} 04:03:02,672 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower751: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower750: synchronized callbacks after 6.288 ms 04:03:02,672 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower751: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=6, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@32343a64} 04:03:02,672 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower751: synchronized callbacks after 5.737 ms 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader749, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): Number of entries to be appended = 1 04:03:02,672 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader749, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): Append entry to log one 04:03:02,672 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): Number of entries to be appended = 1 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower750: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:02,672 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 04:03:02,672 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): Append entry to log one 04:03:02,672 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower751: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): Log size is now 2 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower750, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached -1 (of expected -1) 04:03:02,672 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 374.0 μs 04:03:02,672 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: received 1 commands after 337.6 μs 04:03:02,672 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower750: awaiting more completions to resolve 1 synchronized callback(s) 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower750 updated: matchIndex: 1, nextIndex: 2 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader749 (Leader): Setting last applied to 1 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader749: Moving last applied index from 0 to 1 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader749: Applying state for log index 1 data one 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader749: applyState called: one 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): Log size is now 2 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower751, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower751: Lagging 0 entries behind leader leader749 and reached -1 (of expected -1) 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower751: awaiting more completions to resolve 1 synchronized callback(s) 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - leader749: fakeSnapshot purging log to 0 for term 6 04:03:02,673 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 637.9 μs 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 04:03:02,673 PM [leader749-writer-140] [DEBUG] EntryJournalV1 - leader749: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:03:02,673 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:02,673 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 143.3 μs 04:03:02,673 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: waiting to receive commands 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower751 updated: matchIndex: 1, nextIndex: 2 04:03:02,673 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 04:03:02,673 PM [follower751-writer-139] [DEBUG] RaftStorageCompleter - follower751: 1 completion(s) pending 04:03:02,673 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: completed 1 commands in 614.7 μs 04:03:02,673 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: waiting to receive commands 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower751: running 1 completion(s) 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower751: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower751: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=6, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@448acd23} 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower751: synchronized callbacks after 457.4 μs 04:03:02,673 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): Commit index set to 1 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower751: Lagging 0 entries behind leader leader749 and reached 1 (of expected -1) 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower751 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 1 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower751 (Follower): Setting last applied to 0 04:03:02,673 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower751: Moving last applied index from -1 to 0 04:03:02,673 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 814.7 μs 04:03:02,673 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower751: Applying state for log index 0 data zero 04:03:02,673 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower750: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower751: applyState called: zero 04:03:02,673 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower750: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=6, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@29966bc8} 04:03:02,673 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower750: synchronized callbacks after 884.4 μs 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower751 (Follower): Setting last applied to 1 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower751: Moving last applied index from 0 to 1 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower751: Applying state for log index 1 data one 04:03:02,673 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower751: applyState called: one 04:03:02,673 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): Commit index set to 1 04:03:02,673 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 1 (of expected -1) 04:03:02,674 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 1 04:03:02,674 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower750 (Follower): Setting last applied to 0 04:03:02,674 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower751: fakeSnapshot purging log to 0 for term 6 04:03:02,674 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower750: Moving last applied index from -1 to 0 04:03:02,674 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower750: Applying state for log index 0 data zero 04:03:02,674 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower750: applyState called: zero 04:03:02,674 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: received 1 commands after 404.0 μs 04:03:02,674 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower750 (Follower): Setting last applied to 1 04:03:02,674 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower750: Moving last applied index from 0 to 1 04:03:02,674 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower750: Applying state for log index 1 data one 04:03:02,674 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower750: applyState called: one 04:03:02,674 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower750: fakeSnapshot purging log to 0 for term 6 04:03:02,674 PM [follower751-writer-139] [DEBUG] EntryJournalV1 - follower751: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:03:02,674 PM [follower751-writer-139] [DEBUG] RaftStorageCompleter - follower751: 1 completion(s) pending 04:03:02,674 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 362.7 μs 04:03:02,674 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: completed 1 commands in 149.9 μs 04:03:02,674 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: waiting to receive commands 04:03:02,674 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower751: running 1 completion(s) 04:03:02,674 PM [follower750-writer-138] [DEBUG] EntryJournalV1 - follower750: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:03:02,674 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:02,674 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:02,674 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 116.7 μs 04:03:02,674 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:02,784 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower751: Lagging 0 entries behind leader leader749 and reached 1 (of expected -1) 04:03:02,784 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 1 (of expected -1) 04:03:02,816 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader749: Persist data index=2 term=6 command=two 04:03:02,816 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader749: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:02,816 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 142.6 ms 04:03:02,816 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Replicate message: identifier: MockIdentifier{value=two}, logIndex: 2, isSendImmediate: true 04:03:02,816 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 04:03:02,816 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 2 is present for follower follower751 04:03:02,816 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower751: AppendEntries{term=6, leaderId=leader749, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 04:03:02,816 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 04:03:02,816 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 2 is present for follower follower750 04:03:02,816 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower750: AppendEntries{term=6, leaderId=leader749, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 04:03:02,816 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower750 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader749, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 04:03:02,816 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower750 (Follower): Number of entries to be appended = 1 04:03:02,816 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower750 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 04:03:02,816 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower750 (Follower): Append entry to log two 04:03:02,816 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower750: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:02,817 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower750 (Follower): Log size is now 2 04:03:02,817 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower750 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower750, logLastIndex=2, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:02,817 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 1 (of expected -1) 04:03:02,817 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower750: awaiting more completions to resolve 1 synchronized callback(s) 04:03:02,817 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 142.7 ms 04:03:02,817 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:02,817 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower750 updated: matchIndex: 2, nextIndex: 3 04:03:02,817 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 1.081 ms 04:03:02,817 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: waiting to receive commands 04:03:02,817 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 0 04:03:02,817 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:02,817 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader749: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:02,817 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 04:03:02,817 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader749 (Leader): Setting last applied to 2 04:03:02,817 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader749: Moving last applied index from 1 to 2 04:03:02,817 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader749: Applying state for log index 2 data two 04:03:02,817 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader749: applyState called: two 04:03:02,817 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - leader749: fakeSnapshot purging log to 1 for term 6 04:03:02,817 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 431.8 μs 04:03:02,817 PM [leader749-writer-140] [DEBUG] EntryJournalV1 - leader749: updated meta to JournalMeta[replayFrom=1, applyTo=3] 04:03:02,817 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:02,817 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 180.9 μs 04:03:02,817 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: waiting to receive commands 04:03:02,817 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:02,818 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:02,818 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:02,818 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 1.155 ms 04:03:02,818 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower750: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=false} 04:03:02,818 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:02,818 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower750: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=6, command=two, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@72092770} 04:03:02,818 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower750: synchronized callbacks after 1.319 ms 04:03:02,903 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 2 is present for follower follower751 04:03:02,903 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower751: AppendEntries{term=6, leaderId=leader749, prevLogIndex=1, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}]} 04:03:03,23 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 2 is present for follower follower751 04:03:03,23 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower751: AppendEntries{term=6, leaderId=leader749, prevLogIndex=1, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}]} 04:03:03,24 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): Commit index set to 2 04:03:03,24 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 2 (of expected -1) 04:03:03,24 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower750 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 2 04:03:03,24 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower750 (Follower): Setting last applied to 2 04:03:03,24 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower750: Moving last applied index from 1 to 2 04:03:03,24 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower750: Applying state for log index 2 data two 04:03:03,24 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower750: applyState called: two 04:03:03,24 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower750: fakeSnapshot purging log to 1 for term 6 04:03:03,24 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 206.2 ms 04:03:03,24 PM [follower750-writer-138] [DEBUG] EntryJournalV1 - follower750: updated meta to JournalMeta[replayFrom=1, applyTo=3] 04:03:03,24 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:03,25 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 392.7 μs 04:03:03,25 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:03,25 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader749: Persist data index=3 term=6 command=three 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader749: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Replicate message: identifier: MockIdentifier{value=three}, logIndex: 3, isSendImmediate: true 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: true, followerNextIndex: 2, leaderLastIndex: 3, leaderSnapShotIndex: 1 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 2 is present for follower follower751 04:03:03,67 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 249.7 ms 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 3, leaderLastIndex: 3, leaderSnapShotIndex: 1 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 3 is present for follower follower750 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower750: AppendEntries{term=6, leaderId=leader749, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower750 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader749, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower750 (Follower): Number of entries to be appended = 1 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower750 (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 04:03:03,67 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower750 (Follower): Append entry to log three 04:03:03,68 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower750: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:03,68 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower750 (Follower): Log size is now 2 04:03:03,68 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower750 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower750, logLastIndex=3, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:03,68 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 2 (of expected -1) 04:03:03,68 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower750: awaiting more completions to resolve 1 synchronized callback(s) 04:03:03,68 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 43.09 ms 04:03:03,68 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower750 updated: matchIndex: 3, nextIndex: 4 04:03:03,68 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 4, leaderLastIndex: 3, leaderSnapShotIndex: 1 04:03:03,69 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:03,69 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 1.421 ms 04:03:03,69 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:03,69 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:03,69 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower750: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=false} 04:03:03,69 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower750: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=3, term=6, command=three, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@e6b6739} 04:03:03,69 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower750: synchronized callbacks after 1.609 ms 04:03:03,69 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:03,69 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 2.277 ms 04:03:03,70 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: waiting to receive commands 04:03:03,70 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:03,70 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader749: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:03,70 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader749: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=2, lastAppliedTerm=6, lastIndex=3, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=1, mandatoryTrim=false] 04:03:03,70 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - leader749: captured snapshot at lastSequenceNumber: 3 04:03:03,70 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader749: Persising snapshot at EntryInfo[index=2, term=6]/EntryInfo[index=3, term=6] 04:03:03,70 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader749: support() called 04:03:03,70 PM [leader749-%d628551] [DEBUG] RaftStorage - leader749: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1437-043860DE.tmp 04:03:03,71 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader749: Removed in-memory snapshotted entries, adjusted snapshotIndex: 1 and term: 6 04:03:03,71 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2 04:03:03,71 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader749 (Leader): Setting last applied to 3 04:03:03,71 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader749: Moving last applied index from 2 to 3 04:03:03,71 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader749: Applying state for log index 3 data three 04:03:03,71 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader749: applyState called: three 04:03:03,71 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 1.841 ms 04:03:03,72 PM [leader749-writer-140] [DEBUG] EntryJournalV1 - leader749: updated meta to JournalMeta[replayFrom=1, applyTo=4] 04:03:03,72 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:03,72 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 386.8 μs 04:03:03,72 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: waiting to receive commands 04:03:03,72 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:03,72 PM [leader749-%d628551] [DEBUG] RaftStorage - leader749: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1437-043860DE.v1 04:03:03,73 PM [leader749-%d628551] [DEBUG] RaftStorage - leader749: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1437-043860DE.v1 04:03:03,73 PM [leader749-%d628551] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/TermInfo.properties 04:03:03,73 PM [leader749-%d628551] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/journal-v1-1.log 04:03:03,73 PM [leader749-%d628551] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/metadata-v1-1.log 04:03:03,73 PM [leader749-%d628551] [DEBUG] RaftStorage - leader749: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1436-03564B92.v1 04:03:03,73 PM [leader749-%d628551] [DEBUG] RaftStorage - leader749: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:02.055987090Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1436-03564B92.v1, size=221, entryCompress=NONE, stateCompress=NONE} 04:03:03,73 PM [leader749-%d628551] [DEBUG] RaftStorage - leader749: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:03.070803678Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1437-043860DE.v1, size=374, entryCompress=NONE, stateCompress=NONE} 04:03:03,73 PM [leader749-%d628551] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:03,73 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:03,73 PM [test-pekko.actor.default-dispatcher-17] [INFO] SnapshotManager - leader749: snapshot is durable as of 2025-08-11T16:03:03.070803678Z 04:03:03,73 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SnapshotManager - leader749: Snapshot success, discarding journal entries up to 3 04:03:03,73 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 1.403 ms 04:03:03,73 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorSnapshotMessageSupport - leader749: SnapshotComplete received 04:03:03,73 PM [leader749-writer-140] [DEBUG] EntryJournalV1 - leader749: updated meta to JournalMeta[replayFrom=4, applyTo=4] 04:03:03,73 PM [leader749-writer-140] [DEBUG] EntryJournalV1 - leader749: discarded entries up to (and including) 3 04:03:03,73 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:03,73 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 199.8 μs 04:03:03,74 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: waiting to receive commands 04:03:03,74 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:03,143 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 2 is present for follower follower751 04:03:03,143 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower751: AppendEntries{term=6, leaderId=leader749, prevLogIndex=1, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}, JournaledLogEntry{index=3, term=6, command=three, pending=false}]} 04:03:03,264 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): Commit index set to 3 04:03:03,264 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 3 (of expected -1) 04:03:03,264 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): 2, lastIndex(): 3 04:03:03,264 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower750 (Follower): Setting last applied to 3 04:03:03,264 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower750: Moving last applied index from 2 to 3 04:03:03,264 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower750: Applying state for log index 3 data three 04:03:03,264 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower750: applyState called: three 04:03:03,264 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 194.8 ms 04:03:03,264 PM [follower750-writer-138] [DEBUG] EntryJournalV1 - follower750: updated meta to JournalMeta[replayFrom=1, applyTo=4] 04:03:03,264 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:03,264 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:03,264 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 347.6 μs 04:03:03,264 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:03,268 PM [awaitility-thread] [DEBUG] RaftStorage - leader749: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1437-043860DE.v1 04:03:03,269 PM [awaitility-thread] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/TermInfo.properties 04:03:03,269 PM [awaitility-thread] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/journal-v1-1.log 04:03:03,269 PM [awaitility-thread] [DEBUG] RaftStorage - leader749: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/metadata-v1-1.log 04:03:03,269 PM [awaitility-thread] [DEBUG] RaftStorage - leader749: picked SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:03.070803678Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest2767980028754575397/leader749/snapshot-00000000689A1437-043860DE.v1, size=374, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:03,269 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaAppendEntries: sending 2 more payloads 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader749: Persist data index=4 term=6 command=four 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader749: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:03,270 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 196.3 ms 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Replicate message: identifier: MockIdentifier{value=four}, logIndex: 4, isSendImmediate: true 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: false, followerNextIndex: 2, leaderLastIndex: 4, leaderSnapShotIndex: 1 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 4, leaderLastIndex: 4, leaderSnapShotIndex: 1 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 4 is present for follower follower750 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower750: AppendEntries{term=6, leaderId=leader749, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader749: Persist data index=5 term=6 command=five 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader749: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:03,270 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader749, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Replicate message: identifier: MockIdentifier{value=five}, logIndex: 5, isSendImmediate: true 04:03:03,270 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): Number of entries to be appended = 1 04:03:03,270 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): After cleanup, lastIndex: 3, entries to be added from: 0 04:03:03,270 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): Append entry to log four 04:03:03,270 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: false, followerNextIndex: 2, leaderLastIndex: 5, leaderSnapShotIndex: 1 04:03:03,270 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower750: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:03,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): Log size is now 3 04:03:03,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower750, logLastIndex=4, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:03,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 3 (of expected -1) 04:03:03,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower750: awaiting more completions to resolve 1 synchronized callback(s) 04:03:03,271 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 4, leaderLastIndex: 5, leaderSnapShotIndex: 1 04:03:03,271 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 4 is present for follower follower750 04:03:03,271 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 6.095 ms 04:03:03,271 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower750 updated: matchIndex: 4, nextIndex: 5 04:03:03,271 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 5, leaderLastIndex: 5, leaderSnapShotIndex: 1 04:03:03,271 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 5 is present for follower follower750 04:03:03,271 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower750: AppendEntries{term=6, leaderId=leader749, prevLogIndex=4, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 04:03:03,271 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:03,271 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 1.028 ms 04:03:03,271 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 2.390 μs 04:03:03,271 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:03,271 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader749: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:03,271 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader749 (Leader): Applying to log - commitIndex: 4, lastAppliedIndex: 3 04:03:03,271 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader749 (Leader): Setting last applied to 4 04:03:03,271 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader749: Moving last applied index from 3 to 4 04:03:03,271 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader749: Applying state for log index 4 data four 04:03:03,271 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader749: applyState called: four 04:03:03,272 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:03,272 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 968.7 μs 04:03:03,272 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower750: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=false} 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower750: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=4, term=6, command=four, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@3b19c947} 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower750: synchronized callbacks after 1.120 ms 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader749, prevLogIndex=4, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): Number of entries to be appended = 1 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): After cleanup, lastIndex: 4, entries to be added from: 0 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): Append entry to log five 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower750: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): Log size is now 4 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower750, logLastIndex=5, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 3 (of expected -1) 04:03:03,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower750: awaiting more completions to resolve 1 synchronized callback(s) 04:03:03,272 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 316.9 μs 04:03:03,272 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower750 updated: matchIndex: 5, nextIndex: 6 04:03:03,272 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower750: active: true, followerNextIndex: 6, leaderLastIndex: 5, leaderSnapShotIndex: 1 04:03:03,273 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:03,273 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 763.1 μs 04:03:03,273 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:03,273 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower750: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=false} 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower750: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=5, term=6, command=five, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@523c2871} 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower750: synchronized callbacks after 875.8 μs 04:03:03,273 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 1.912 ms 04:03:03,273 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 5.950 μs 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): Commit index set to 4 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 4 (of expected -1) 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 4, context.getLastApplied(): 3, lastIndex(): 5 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower750 (Follower): Setting last applied to 4 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower750: Moving last applied index from 3 to 4 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower750: Applying state for log index 4 data four 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower750: applyState called: four 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader749: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader749 (Leader): Applying to log - commitIndex: 5, lastAppliedIndex: 4 04:03:03,273 PM [leader749-writer-140] [DEBUG] EntryJournalV1 - leader749: updated meta to JournalMeta[replayFrom=4, applyTo=5] 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader749 (Leader): Setting last applied to 5 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader749: Moving last applied index from 4 to 5 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader749: Applying state for log index 5 data five 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader749: applyState called: five 04:03:03,273 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:03,273 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 278.8 μs 04:03:03,273 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 3.670 μs 04:03:03,273 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 480.4 μs 04:03:03,273 PM [leader749-writer-140] [DEBUG] EntryJournalV1 - leader749: updated meta to JournalMeta[replayFrom=4, applyTo=6] 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): Commit index set to 5 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 5 (of expected -1) 04:03:03,273 PM [leader749-writer-140] [DEBUG] RaftStorageCompleter - leader749: 1 completion(s) pending 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower750 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 5, context.getLastApplied(): 4, lastIndex(): 5 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower750 (Follower): Setting last applied to 5 04:03:03,273 PM [follower750-writer-138] [DEBUG] EntryJournalV1 - follower750: updated meta to JournalMeta[replayFrom=1, applyTo=5] 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower750: Moving last applied index from 4 to 5 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower750: Applying state for log index 5 data five 04:03:03,273 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower750: applyState called: five 04:03:03,273 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 157.1 μs 04:03:03,273 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 138.6 μs 04:03:03,273 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: waiting to receive commands 04:03:03,273 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:03,273 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:03,274 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader749: running 1 completion(s) 04:03:03,274 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 93.87 μs 04:03:03,274 PM [follower750-writer-138] [DEBUG] EntryJournalV1 - follower750: updated meta to JournalMeta[replayFrom=1, applyTo=6] 04:03:03,274 PM [follower750-writer-138] [DEBUG] RaftStorageCompleter - follower750: 1 completion(s) pending 04:03:03,274 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 180.2 μs 04:03:03,274 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: waiting to receive commands 04:03:03,274 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower750: running 1 completion(s) 04:03:03,384 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 5 (of expected -1) 04:03:03,384 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower751: Lagging -2 entries behind leader leader749 and reached 1 (of expected -1) 04:03:03,384 PM [test-pekko.actor.default-dispatcher-16] [WARN] AbstractLeader - leader749 (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply{term=6, success=true, followerId=follower751, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, timeSinceLastActivity : 599, lastApplied : 5, commitIndex : 5 04:03:03,384 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader749 (Leader): sendAppendEntries: 2 is present for follower follower751 04:03:03,384 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader749 (Leader): Sending AppendEntries to follower follower751: AppendEntries{term=6, leaderId=leader749, prevLogIndex=1, prevLogTerm=6, leaderCommit=5, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}, JournaledLogEntry{index=3, term=6, command=three, pending=false}, JournaledLogEntry{index=4, term=6, command=four, pending=false}, JournaledLogEntry{index=5, term=6, command=five, pending=false}]} 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader749, prevLogIndex=1, prevLogTerm=6, leaderCommit=5, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=false}, JournaledLogEntry{index=3, term=6, command=three, pending=false}, JournaledLogEntry{index=4, term=6, command=four, pending=false}, JournaledLogEntry{index=5, term=6, command=five, pending=false}]} 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): Number of entries to be appended = 4 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): Append entry to log two 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower751: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=false} 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): Append entry to log three 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower751: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=false} 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): Append entry to log four 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower751: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=false} 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): Append entry to log five 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower751: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=false} 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): Log size is now 5 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): Commit index set to 5 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower751, logLastIndex=5, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:03,385 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: received 2 commands after 711.1 ms 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower751: Lagging 0 entries behind leader leader749 and reached 5 (of expected -1) 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower751 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 5, context.getLastApplied(): 1, lastIndex(): 5 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower751 (Follower): Setting last applied to 2 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower751: Moving last applied index from 1 to 2 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower751: Applying state for log index 2 data two 04:03:03,385 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader749 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower751 updated: matchIndex: 5, nextIndex: 6 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower751: applyState called: two 04:03:03,385 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader749: fakeSnapshot purging log to 4 for term 6 04:03:03,385 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader749 (Leader): Checking sendAppendEntries for follower follower751: active: true, followerNextIndex: 6, leaderLastIndex: 5, leaderSnapShotIndex: 4 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower751 (Follower): Setting last applied to 3 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower751: Moving last applied index from 2 to 3 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower751: Applying state for log index 3 data three 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower751: applyState called: three 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower751 (Follower): Setting last applied to 4 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower751: Moving last applied index from 3 to 4 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower751: Applying state for log index 4 data four 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower751: applyState called: four 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower751 (Follower): Setting last applied to 5 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower751: Moving last applied index from 4 to 5 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower751: Applying state for log index 5 data five 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower751: applyState called: five 04:03:03,385 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower751: fakeSnapshot purging log to 1 for term 6 04:03:03,386 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower751: awaiting more completions to resolve 4 synchronized callback(s) 04:03:03,387 PM [follower751-writer-139] [DEBUG] RaftStorageCompleter - follower751: 2 completion(s) pending 04:03:03,387 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: completed 2 commands in 1.772 ms 04:03:03,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower751: running 2 completion(s) 04:03:03,387 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: received 3 commands after 3.240 μs 04:03:03,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower751: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=false} 04:03:03,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower751: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=6, command=two, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@354be127} 04:03:03,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower751: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=false} 04:03:03,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower751: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=3, term=6, command=three, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@354be127} 04:03:03,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower751: awaiting more completions to resolve 2 synchronized callback(s) 04:03:03,389 PM [follower751-writer-139] [DEBUG] EntryJournalV1 - follower751: updated meta to JournalMeta[replayFrom=1, applyTo=6] 04:03:03,389 PM [follower751-writer-139] [DEBUG] RaftStorageCompleter - follower751: 3 completion(s) pending 04:03:03,389 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: completed 3 commands in 1.831 ms 04:03:03,389 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: waiting to receive commands 04:03:03,389 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower751: running 3 completion(s) 04:03:03,389 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower751: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=false} 04:03:03,389 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower751: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=4, term=6, command=four, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@354be127} 04:03:03,389 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower751: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=false} 04:03:03,389 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower751: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=5, term=6, command=five, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@354be127} 04:03:03,389 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower751: synchronized callbacks after 3.199 ms 04:03:03,503 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 5 (of expected -1) 04:03:03,503 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower750: fakeSnapshot purging log to 4 for term 6 04:03:03,503 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower751: Lagging 0 entries behind leader leader749 and reached 5 (of expected -1) 04:03:03,504 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SnapshotManager - follower751: fakeSnapshot purging log to 4 for term 6 04:03:03,624 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower751: Lagging 0 entries behind leader leader749 and reached 5 (of expected -1) 04:03:03,624 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 5 (of expected -1) 04:03:03,743 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower751: Lagging 0 entries behind leader leader749 and reached 5 (of expected -1) 04:03:03,743 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower750: Lagging 0 entries behind leader leader749 and reached 5 (of expected -1) 04:03:03,775 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaAppendEntries complete 04:03:03,775 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower750-collector752#1578617081] 04:03:03,777 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower750] 04:03:03,777 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EnabledRaftStorage - follower750: terminating thread VirtualThread[#920,follower750-writer-138]/waiting 04:03:03,777 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: received 1 commands after 503.5 ms 04:03:03,778 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: completed 1 commands in 21.38 μs 04:03:03,778 PM [follower750-writer-138] [DEBUG] JournalWriteTask - follower750: journal writer stopped 04:03:03,778 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower750: journal closed 04:03:03,778 PM [test-pekko.actor.default-dispatcher-7] [INFO] EnabledRaftStorage - follower750: journal closed 04:03:03,778 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower750: stopped executor with 0 remaining tasks 04:03:03,778 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower750: stopped executor with 0 remaining tasks 04:03:03,778 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower751-collector753#219554506] 04:03:03,778 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower751] 04:03:03,778 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EnabledRaftStorage - follower751: terminating thread VirtualThread[#921,follower751-writer-139]/waiting 04:03:03,779 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: received 1 commands after 389.9 ms 04:03:03,779 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: completed 1 commands in 9.580 μs 04:03:03,779 PM [follower751-writer-139] [DEBUG] JournalWriteTask - follower751: journal writer stopped 04:03:03,779 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower751: journal closed 04:03:03,779 PM [test-pekko.actor.default-dispatcher-7] [INFO] EnabledRaftStorage - follower751: journal closed 04:03:03,779 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower751: stopped executor with 0 remaining tasks 04:03:03,779 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower751: stopped executor with 0 remaining tasks 04:03:03,779 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader749-collector754#1332676326] 04:03:03,779 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader749] 04:03:03,779 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EnabledRaftStorage - leader749: terminating thread VirtualThread[#922,leader749-writer-140]/waiting 04:03:03,779 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: received 1 commands after 506.0 ms 04:03:03,779 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: completed 1 commands in 8.981 μs 04:03:03,779 PM [leader749-writer-140] [DEBUG] JournalWriteTask - leader749: journal writer stopped 04:03:03,780 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - leader749: journal closed 04:03:03,780 PM [test-pekko.actor.default-dispatcher-7] [INFO] EnabledRaftStorage - leader749: journal closed 04:03:03,780 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: stopped executor with 0 remaining tasks 04:03:03,780 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader749: stopped executor with 0 remaining tasks 04:03:03,782 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderInstallsSnapshotWithRestartedFollowerDuringSnapshotInstallation starting [WARN] [08/11/2025 16:03:03.784] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower759]: org.opendaylight.controller.cluster.common.actor.Monitor@5f09b1af 04:03:03,785 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActor - follower759: Starting recovery with journal batch size 1000 04:03:03,785 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower759: started executor 04:03:03,785 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower759: started executor 04:03:03,786 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower759: metadata journal open: replayFrom=1 applyTo=0 04:03:03,787 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower759: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:03,787 PM [test-pekko.actor.default-dispatcher-7] [INFO] EnabledRaftStorage - follower759: journal open: applyTo=0 04:03:03,787 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: journal writer started 04:03:03,787 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:03,788 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower759/TermInfo.properties does not exist 04:03:03,788 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower759: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower759/journal-v1-1.log [WARN] [08/11/2025 16:03:03.788] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower760]: org.opendaylight.controller.cluster.common.actor.Monitor@279a8080 04:03:03,788 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower759: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower759/metadata-v1-1.log 04:03:03,788 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower759: no eligible files found 04:03:03,788 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower759: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:03,788 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower760: Starting recovery with journal batch size 1000 04:03:03,788 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower760: started executor 04:03:03,788 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower760: started executor 04:03:03,790 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower760: metadata journal open: replayFrom=1 applyTo=0 04:03:03,790 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower760: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:03,790 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - follower760: journal open: applyTo=0 04:03:03,790 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: journal writer started 04:03:03,790 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: waiting to receive commands 04:03:03,791 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties does not exist 04:03:03,791 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:03,791 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:03,791 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:03,791 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower760: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [08/11/2025 16:03:03.791] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader758]: org.opendaylight.controller.cluster.common.actor.Monitor@51fd9248 04:03:03,792 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - leader758: Starting recovery with journal batch size 1000 04:03:03,792 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader758: started executor 04:03:03,792 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader758: started executor 04:03:03,792 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader758: metadata journal open: replayFrom=1 applyTo=0 04:03:03,792 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader758: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:03,792 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - leader758: journal open: applyTo=0 04:03:03,793 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: journal writer started 04:03:03,793 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:03,794 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/TermInfo.properties does not exist 04:03:03,794 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/journal-v1-1.log 04:03:03,794 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/metadata-v1-1.log 04:03:03,794 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader758: no eligible files found 04:03:03,794 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader758: Set currentTerm=5, votedFor=leader758 04:03:03,794 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader758: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:03,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower759: Set currentTerm=0, votedFor=null 04:03:03,799 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower759: Local TermInfo store seeded with TermInfo{term=0} 04:03:03,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower759: Pekko recovery completed and can restore from snapshot 04:03:03,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower759: starting journal recovery 04:03:03,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower759: Moving last applied index from -1 to -1 04:03:03,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower759: empty journal: appending 0 entries 04:03:03,800 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower759: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:03,800 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower759: journal recovery completedin 349.7 μs with journalIndex=1 04:03:03,800 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower759: Journal recovery completed and can restore from snapshot 04:03:03,805 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader758: Set currentTerm=5, votedFor=leader758 04:03:03,805 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader758: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader758} 04:03:03,805 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader758: taking snapshot to clear Pekko persistence to 1 04:03:03,805 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Recovery - leader758: no applied entries in recovery log, re-snapshotting EntryInfo[index=-1, term=-1] 04:03:03,805 PM [test-pekko.actor.default-dispatcher-15] [INFO] Recovery - leader758: Taking snapshot on entry with index -1 04:03:03,805 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader758: support() called 04:03:03,806 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader758: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1437-300B602C.tmp 04:03:03,806 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower760: Set currentTerm=0, votedFor=null 04:03:03,806 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower760: Local TermInfo store seeded with TermInfo{term=0} 04:03:03,806 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower760: Pekko recovery completed and can restore from snapshot 04:03:03,806 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower760: starting journal recovery 04:03:03,806 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower760: Moving last applied index from -1 to -1 04:03:03,806 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower760: empty journal: appending 0 entries 04:03:03,806 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower760: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:03,806 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower760: journal recovery completedin 361.9 μs with journalIndex=1 04:03:03,806 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower760: Journal recovery completed and can restore from snapshot 04:03:03,815 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader758: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1437-300B602C.v1 04:03:03,815 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader758: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1437-300B602C.v1 04:03:03,815 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/TermInfo.properties 04:03:03,815 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/journal-v1-1.log 04:03:03,815 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/metadata-v1-1.log 04:03:03,815 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader758: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:03.806051884Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1437-300B602C.v1, size=221, entryCompress=NONE, stateCompress=NONE} 04:03:03,816 PM [test-pekko.actor.default-dispatcher-15] [INFO] Recovery - leader758: Snapshot completed in 10.35 ms, resetting timer for the next recovery snapshot 04:03:03,816 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader758: Pekko recovery completed and cannot restore from snapshot 04:03:03,816 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader758: starting journal recovery 04:03:03,816 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader758: Moving last applied index from -1 to -1 04:03:03,816 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - leader758: empty journal: appending 0 entries 04:03:03,816 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:03,816 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - leader758: journal recovery completedin 376.8 μs with journalIndex=1 04:03:03,816 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader758: Journal recovery completed and cannot restore from snapshot 04:03:03,817 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader758: ignoring DeleteMessagesSuccess(1) 04:03:04,243 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 04:03:04,243 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - leader758 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.10:17355, Removed) 04:03:04,243 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - leader758 (Follower): Received ElectionTimeout - switching to Candidate 04:03:04,243 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader758 (Candidate): Election: Candidate has following voting peers: [follower759, follower760] 04:03:04,248 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader758: Set currentTerm=6, votedFor=leader758 04:03:04,248 PM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader758 (Candidate): Starting new election term 6 04:03:04,248 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader758 (Candidate): Sending RequestVote{term=6, candidateId=leader758, lastLogIndex=-1, lastLogTerm=-1} to peer follower759 04:03:04,248 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader758 (Candidate): Sending RequestVote{term=6, candidateId=leader758, lastLogIndex=-1, lastLogTerm=-1} to peer follower760 04:03:04,248 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower759 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 04:03:04,248 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - leader758 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 04:03:04,248 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower759 (Follower): Cluster state: [] 04:03:04,248 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower759 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader758, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 04:03:04,248 PM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower759 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader758, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 04:03:04,248 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower760 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 04:03:04,248 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower760 (Follower): Cluster state: [] 04:03:04,248 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower760 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader758, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 04:03:04,248 PM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - follower760 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader758, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 04:03:04,252 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower759: Set currentTerm=6, votedFor=null 04:03:04,252 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower759 (Follower): In requestVote: RequestVote{term=6, candidateId=leader758, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 04:03:04,252 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower760: Set currentTerm=6, votedFor=null 04:03:04,253 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower760 (Follower): In requestVote: RequestVote{term=6, candidateId=leader758, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 04:03:04,255 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower759: Set currentTerm=6, votedFor=leader758 04:03:04,255 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower759 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 04:03:04,255 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader758 (Candidate): RaftRPC message received RequestVoteReply{term=6, voteGranted=true}, my term is 6 04:03:04,255 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader758 (Candidate): handleRequestVoteReply: RequestVoteReply{term=6, voteGranted=true}, current voteCount: 1 04:03:04,255 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader758 (Leader): Election: Leader has following peers: [follower759, follower760] 04:03:04,255 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:04,255 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:04,255 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader758 (Candidate) :- Switching from behavior Candidate to Leader, election term: 6 04:03:04,255 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower760: Set currentTerm=6, votedFor=leader758 04:03:04,255 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower759: Last sync leader does not match current leader leader758, need to catch up to -1 04:03:04,255 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower760 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 04:03:04,256 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower759 updated: matchIndex: -1, nextIndex: 0 04:03:04,256 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower760: Last sync leader does not match current leader leader758, need to catch up to -1 [WARN] [08/11/2025 16:03:04.256] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower759] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower759, initialSyncDone=false] [WARN] [08/11/2025 16:03:04.256] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower760] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower760, initialSyncDone=false] 04:03:04,256 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:04,256 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower760 updated: matchIndex: -1, nextIndex: 0 04:03:04,256 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:04,270 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - Leader created and elected 04:03:04,270 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader758: Persist data index=0 term=6 command=zero 04:03:04,270 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader758: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:04,271 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 477.9 ms 04:03:04,271 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 04:03:04,271 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:04,271 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 0 is present for follower follower759 04:03:04,271 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Sending AppendEntries to follower follower759: AppendEntries{term=6, leaderId=leader758, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:04,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader758, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:04,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Number of entries to be appended = 1 04:03:04,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 04:03:04,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Append entry to log zero 04:03:04,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:04,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Log size is now 1 04:03:04,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower759, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:04,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached -1 (of expected -1) 04:03:04,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: awaiting more completions to resolve 1 synchronized callback(s) 04:03:04,272 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 484.9 ms 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 0 is present for follower follower760 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Sending AppendEntries to follower follower760: AppendEntries{term=6, leaderId=leader758, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower759 updated: matchIndex: 0, nextIndex: 1 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:04,272 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader758, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader758: Persist data index=1 term=6 command=one 04:03:04,272 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): Number of entries to be appended = 1 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader758: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:03:04,272 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 04:03:04,272 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): Append entry to log zero 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 04:03:04,272 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower760: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 1 is present for follower follower759 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Sending AppendEntries to follower follower759: AppendEntries{term=6, leaderId=leader758, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:03:04,272 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): Log size is now 1 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:04,272 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 0 is present for follower follower760 04:03:04,272 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower760, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:04,272 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower760: Lagging 0 entries behind leader leader758 and reached -1 (of expected -1) 04:03:04,272 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower760: awaiting more completions to resolve 1 synchronized callback(s) 04:03:04,272 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: received 1 commands after 482.0 ms 04:03:04,273 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower760 updated: matchIndex: 0, nextIndex: 1 04:03:04,273 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:04,273 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 1 is present for follower follower760 04:03:04,273 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Sending AppendEntries to follower follower760: AppendEntries{term=6, leaderId=leader758, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:03:04,276 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:04,276 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:04,276 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 5.066 ms 04:03:04,276 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader758: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:04,276 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 7.200 μs 04:03:04,276 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 04:03:04,276 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader758 (Leader): Setting last applied to 0 04:03:04,276 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader758: Moving last applied index from -1 to 0 04:03:04,276 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader758: Applying state for log index 0 data zero 04:03:04,276 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader758: applyState called: zero 04:03:04,277 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:04,277 PM [follower760-writer-142] [DEBUG] RaftStorageCompleter - follower760: 1 completion(s) pending 04:03:04,277 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:04,277 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 1.510 ms 04:03:04,277 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 3.440 μs 04:03:04,277 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 5.704 ms 04:03:04,277 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: completed 1 commands in 4.813 ms 04:03:04,277 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:04,277 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: waiting to receive commands 04:03:04,277 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: running 1 completion(s) 04:03:04,277 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower760: running 1 completion(s) 04:03:04,277 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower760: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=6, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@3cf3b7f7} 04:03:04,278 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=1, applyTo=1] 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader758: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: synchronized callbacks after 6.030 ms 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower760: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=6, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@74411c3d} 04:03:04,278 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower760: synchronized callbacks after 5.123 ms 04:03:04,278 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 247.1 μs 04:03:04,278 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader758, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Number of entries to be appended = 1 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Append entry to log one 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader758, prevLogIndex=0, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=false}]} 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): Number of entries to be appended = 1 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Log size is now 2 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower759, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): Append entry to log one 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower760: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:04,278 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 463.7 μs 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached -1 (of expected -1) 04:03:04,278 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: awaiting more completions to resolve 1 synchronized callback(s) 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower759 updated: matchIndex: 1, nextIndex: 2 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 04:03:04,278 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: received 1 commands after 548.7 μs 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - leader758 (Leader): Setting last applied to 1 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader758: Moving last applied index from 0 to 1 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader758: Applying state for log index 1 data one 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader758: applyState called: one 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): Log size is now 2 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower760, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower760: Lagging 0 entries behind leader leader758 and reached -1 (of expected -1) 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - leader758: fakeSnapshot purging log to 0 for term 6 04:03:04,278 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower760: awaiting more completions to resolve 1 synchronized callback(s) 04:03:04,278 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 652.7 μs 04:03:04,278 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 04:03:04,278 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:03:04,279 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:04,279 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:04,279 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 200.0 μs 04:03:04,279 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:04,279 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower760 updated: matchIndex: 1, nextIndex: 2 04:03:04,279 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 04:03:04,279 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:04,279 PM [follower760-writer-142] [DEBUG] RaftStorageCompleter - follower760: 1 completion(s) pending 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: running 1 completion(s) 04:03:04,279 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: completed 1 commands in 734.1 μs 04:03:04,279 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 837.8 μs 04:03:04,279 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: waiting to receive commands 04:03:04,279 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=6, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@6207ada6} 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower760: running 1 completion(s) 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower760: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: synchronized callbacks after 933.0 μs 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower760: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=6, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@564848ee} 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower760: synchronized callbacks after 610.7 μs 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Commit index set to 1 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): Commit index set to 1 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 1 (of expected -1) 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 1 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower760: Lagging 0 entries behind leader leader758 and reached 1 (of expected -1) 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower759 (Follower): Setting last applied to 0 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower760 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 1 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower759: Moving last applied index from -1 to 0 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower760 (Follower): Setting last applied to 0 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower759: Applying state for log index 0 data zero 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower760: Moving last applied index from -1 to 0 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower759: applyState called: zero 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower760: Applying state for log index 0 data zero 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower760: applyState called: zero 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower759 (Follower): Setting last applied to 1 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower760 (Follower): Setting last applied to 1 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower759: Moving last applied index from 0 to 1 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower760: Moving last applied index from 0 to 1 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower760: Applying state for log index 1 data one 04:03:04,279 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower760: applyState called: one 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower759: Applying state for log index 1 data one 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower759: applyState called: one 04:03:04,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower759: fakeSnapshot purging log to 0 for term 6 04:03:04,279 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: received 1 commands after 590.4 μs 04:03:04,279 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 594.1 μs [WARN] [08/11/2025 16:03:04.279] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower759] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower759, initialSyncDone=true] 04:03:04,279 PM [follower759-writer-141] [DEBUG] EntryJournalV1 - follower759: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:03:04,280 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:04,280 PM [follower760-writer-142] [DEBUG] EntryJournalV1 - follower760: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:03:04,280 PM [follower760-writer-142] [DEBUG] RaftStorageCompleter - follower760: 1 completion(s) pending 04:03:04,280 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 157.6 μs 04:03:04,280 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: running 1 completion(s) 04:03:04,280 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:04,280 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: completed 1 commands in 209.3 μs 04:03:04,280 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: waiting to receive commands 04:03:04,280 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower760: fakeSnapshot purging log to 0 for term 6 04:03:04,280 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower760: running 1 completion(s) [WARN] [08/11/2025 16:03:04.280] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower760] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower760, initialSyncDone=true] 04:03:04,493 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower760: Lagging 0 entries behind leader leader758 and reached 1 (of expected -1) 04:03:04,494 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 1 (of expected -1) 04:03:04,523 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EnabledRaftStorage - follower760: terminating thread VirtualThread[#929,follower760-writer-142]/waiting 04:03:04,523 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: received 1 commands after 243.3 ms 04:03:04,523 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: completed 1 commands in 22.15 μs 04:03:04,523 PM [follower760-writer-142] [DEBUG] JournalWriteTask - follower760: journal writer stopped 04:03:04,523 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower760: journal closed 04:03:04,523 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - follower760: journal closed 04:03:04,524 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower760: stopped executor with 0 remaining tasks 04:03:04,524 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower760: stopped executor with 0 remaining tasks 04:03:04,614 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 1 (of expected -1) [WARN] [08/11/2025 16:03:04.614] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [08/11/2025 16:03:04.734] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:04,734 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 1 (of expected -1) [WARN] [08/11/2025 16:03:04.854] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:04,855 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 1 (of expected -1) 04:03:04,928 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader758: Persist data index=2 term=6 command=two 04:03:04,928 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader758: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:04,928 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 649.4 ms 04:03:04,928 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Replicate message: identifier: MockIdentifier{value=two}, logIndex: 2, isSendImmediate: true 04:03:04,928 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 04:03:04,928 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 2 is present for follower follower759 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Sending AppendEntries to follower follower759: AppendEntries{term=6, leaderId=leader758, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: false, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 04:03:04,929 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader758, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 04:03:04,929 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Number of entries to be appended = 1 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader758: Persist data index=3 term=6 command=three 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader758: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Replicate message: identifier: MockIdentifier{value=three}, logIndex: 3, isSendImmediate: true [WARN] [08/11/2025 16:03:04.929] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:04,929 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 2, leaderLastIndex: 3, leaderSnapShotIndex: 0 04:03:04,929 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Append entry to log two 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 2 is present for follower follower759 04:03:04,929 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: false, followerNextIndex: 2, leaderLastIndex: 3, leaderSnapShotIndex: 0 04:03:04,929 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 649.5 ms 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader758: Persist data index=4 term=6 command=four 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader758: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Replicate message: identifier: MockIdentifier{value=four}, logIndex: 4, isSendImmediate: true [WARN] [08/11/2025 16:03:04.929] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 2, leaderLastIndex: 4, leaderSnapShotIndex: 0 04:03:04,929 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Log size is now 2 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 2 is present for follower follower759 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: false, followerNextIndex: 2, leaderLastIndex: 4, leaderSnapShotIndex: 0 04:03:04,929 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower759, logLastIndex=2, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:04,929 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 1 (of expected -1) 04:03:04,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower759 updated: matchIndex: 2, nextIndex: 3 [WARN] [08/11/2025 16:03:04.929] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 3, leaderLastIndex: 4, leaderSnapShotIndex: 0 04:03:04,930 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: awaiting more completions to resolve 1 synchronized callback(s) 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 3 is present for follower follower759 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Sending AppendEntries to follower follower759: AppendEntries{term=6, leaderId=leader758, prevLogIndex=2, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}, JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader758: Persist data index=5 term=6 command=five 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader758: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Replicate message: identifier: MockIdentifier{value=five}, logIndex: 5, isSendImmediate: true 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 3, leaderLastIndex: 5, leaderSnapShotIndex: 0 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 3 is present for follower follower759 04:03:04,930 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:04,930 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 1.848 ms 04:03:04,930 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 3 commands after 4.120 μs 04:03:04,930 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: false, followerNextIndex: 2, leaderLastIndex: 5, leaderSnapShotIndex: 0 04:03:04,930 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: running 1 completion(s) 04:03:04,930 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 1.027 ms 04:03:04,930 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:04,930 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=false} 04:03:04,930 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=6, command=two, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@21f813de} 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader758: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=true} [WARN] [08/11/2025 16:03:04.930] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 04:03:04,930 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: synchronized callbacks after 740.4 μs 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader758 (Leader): Setting last applied to 2 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader758: Moving last applied index from 1 to 2 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader758: Applying state for log index 2 data two 04:03:04,930 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader758: applyState called: two 04:03:04,930 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader758, prevLogIndex=2, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}, JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 04:03:04,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Number of entries to be appended = 2 04:03:04,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 04:03:04,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Append entry to log three 04:03:04,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:04,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Append entry to log four 04:03:04,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:04,931 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 441.0 μs 04:03:04,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Log size is now 4 04:03:04,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower759, logLastIndex=4, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:04,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader758: fakeSnapshot purging log to 1 for term 6 04:03:04,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 1 (of expected -1) 04:03:04,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower759 updated: matchIndex: 4, nextIndex: 5 04:03:04,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: awaiting more completions to resolve 2 synchronized callback(s) 04:03:04,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 5, leaderLastIndex: 5, leaderSnapShotIndex: 1 04:03:04,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 5 is present for follower follower759 04:03:04,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Sending AppendEntries to follower follower759: AppendEntries{term=6, leaderId=leader758, prevLogIndex=4, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 04:03:04,931 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:04,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader758: Persist data index=6 term=6 command=six 04:03:04,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader758: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=true} 04:03:04,931 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 736.3 μs 04:03:04,931 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 2.200 μs 04:03:04,931 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Replicate message: identifier: MockIdentifier{value=six}, logIndex: 6, isSendImmediate: true 04:03:04,932 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 5, leaderLastIndex: 6, leaderSnapShotIndex: 1 04:03:04,932 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 5 is present for follower follower759 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: running 1 completion(s) 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=false} 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=3, term=6, command=three, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@da94772} 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: awaiting more completions to resolve 1 synchronized callback(s) 04:03:04,932 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: false, followerNextIndex: 2, leaderLastIndex: 6, leaderSnapShotIndex: 1 04:03:04,932 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:04,932 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 651.9 μs 04:03:04,932 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands [WARN] [08/11/2025 16:03:04.932] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=1, prevLogTerm=6, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: running 1 completion(s) 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=false} 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=4, term=6, command=four, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@da94772} 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: synchronized callbacks after 1.129 ms 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader758, prevLogIndex=4, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Number of entries to be appended = 1 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): After cleanup, lastIndex: 4, entries to be added from: 0 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Append entry to log five 04:03:04,932 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:04,933 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Log size is now 5 04:03:04,933 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 396.1 μs 04:03:04,933 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Commit index set to 2 04:03:04,933 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower759, logLastIndex=5, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:04,933 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 3 completion(s) pending 04:03:04,933 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower759 updated: matchIndex: 5, nextIndex: 6 04:03:04,933 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 3 commands in 2.659 ms 04:03:04,933 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 2 commands after 3.850 μs 04:03:04,933 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 6, leaderLastIndex: 6, leaderSnapShotIndex: 1 04:03:04,933 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 6 is present for follower follower759 04:03:04,933 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): Sending AppendEntries to follower follower759: AppendEntries{term=6, leaderId=leader758, prevLogIndex=5, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 04:03:04,933 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=1, applyTo=3] 04:03:04,933 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - leader758: running 3 completion(s) 04:03:04,933 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader758: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:04,933 PM [test-pekko.actor.default-dispatcher-17] [INFO] SnapshotManager - leader758: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=2, lastAppliedTerm=6, lastIndex=3, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=4, mandatoryTrim=false] 04:03:04,933 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:04,933 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SnapshotManager - leader758: captured snapshot at lastSequenceNumber: 3 04:03:04,933 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 619.3 μs 04:03:04,933 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:04,933 PM [test-pekko.actor.default-dispatcher-17] [INFO] SnapshotManager - leader758: Persising snapshot at EntryInfo[index=2, term=6]/EntryInfo[index=3, term=6] 04:03:04,933 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader758: support() called 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SnapshotManager - leader758: log size 5 exceeds the snapshot batch count 4 - doing snapshotPreCommit with index 2 04:03:04,934 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 2 completion(s) pending 04:03:04,934 PM [leader758-%d210633] [DEBUG] RaftStorage - leader758: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1438-37ADBEA0.tmp 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [INFO] SnapshotManager - leader758: Removed in-memory snapshotted entries, adjusted snapshotIndex: 2 and term: 6 04:03:04,934 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 2 commands in 1.068 ms 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2 04:03:04,934 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - leader758 (Leader): Setting last applied to 3 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader758: Moving last applied index from 2 to 3 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader758: Applying state for log index 3 data three 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader758: applyState called: three 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader758: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): Applying to log - commitIndex: 4, lastAppliedIndex: 3 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - leader758 (Leader): Setting last applied to 4 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader758: Moving last applied index from 3 to 4 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader758: Applying state for log index 4 data four 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader758: applyState called: four 04:03:04,934 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 246.3 μs 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader758: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): Applying to log - commitIndex: 5, lastAppliedIndex: 4 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - leader758 (Leader): Setting last applied to 5 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader758: Moving last applied index from 4 to 5 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader758: Applying state for log index 5 data five 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader758: applyState called: five 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - leader758: running 2 completion(s) 04:03:04,934 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader758: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=true} 04:03:04,934 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=1, applyTo=4] 04:03:04,934 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:04,934 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 317.8 μs 04:03:04,935 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:04,935 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 2 commands after 3.280 μs 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 2 (of expected -1) 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 5 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower759 (Follower): Setting last applied to 2 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower759: Moving last applied index from 1 to 2 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower759: Applying state for log index 2 data two 04:03:04,935 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=1, applyTo=5] 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower759: applyState called: two 04:03:04,935 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=1, applyTo=6] 04:03:04,935 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 2 completion(s) pending 04:03:04,935 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 2 commands in 196.5 μs 04:03:04,935 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:04,935 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 1.598 ms 04:03:04,935 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader758: running 2 completion(s) 04:03:04,935 PM [follower759-writer-141] [DEBUG] EntryJournalV1 - follower759: updated meta to JournalMeta[replayFrom=1, applyTo=3] 04:03:04,935 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 more completion(s) pending 04:03:04,935 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 234.6 μs 04:03:04,935 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower759: fakeSnapshot purging log to 1 for term 6 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: running 2 completion(s) 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=false} 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=5, term=6, command=five, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@338543fa} 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: synchronized callbacks after 66.44 μs 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader758, prevLogIndex=5, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=6, term=6, command=six, pending=false}]} 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Number of entries to be appended = 1 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): After cleanup, lastIndex: 5, entries to be added from: 0 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Append entry to log six 04:03:04,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=false} 04:03:04,935 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 400.1 μs 04:03:04,936 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): Log size is now 5 04:03:04,936 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower759 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower759, logLastIndex=6, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:04,936 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 2 (of expected -1) 04:03:04,936 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: awaiting more completions to resolve 1 synchronized callback(s) 04:03:04,936 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower759 updated: matchIndex: 6, nextIndex: 7 04:03:04,936 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader758 (Leader): Applying to log - commitIndex: 6, lastAppliedIndex: 5 04:03:04,936 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader758 (Leader): Setting last applied to 6 04:03:04,936 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader758: Moving last applied index from 5 to 6 04:03:04,936 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader758: Applying state for log index 6 data six 04:03:04,936 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader758: applyState called: six 04:03:04,936 PM [leader758-%d210633] [DEBUG] RaftStorage - leader758: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1438-37ADBEA0.v1 04:03:04,936 PM [leader758-%d210633] [DEBUG] RaftStorage - leader758: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1437-300B602C.v1 04:03:04,936 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:04,936 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 671.3 μs 04:03:04,936 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:04,936 PM [leader758-%d210633] [DEBUG] RaftStorage - leader758: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1438-37ADBEA0.v1 04:03:04,936 PM [leader758-%d210633] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/TermInfo.properties 04:03:04,936 PM [leader758-%d210633] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/journal-v1-1.log 04:03:04,936 PM [leader758-%d210633] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/metadata-v1-1.log 04:03:04,936 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 1.484 ms 04:03:04,936 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: running 1 completion(s) 04:03:04,936 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower759: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=false} 04:03:04,936 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=6, term=6, command=six, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@7e083598} 04:03:04,936 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: synchronized callbacks after 578.8 μs 04:03:04,936 PM [leader758-%d210633] [DEBUG] RaftStorage - leader758: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:03.806051884Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1437-300B602C.v1, size=221, entryCompress=NONE, stateCompress=NONE} 04:03:04,936 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=1, applyTo=7] 04:03:04,936 PM [leader758-%d210633] [DEBUG] RaftStorage - leader758: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:04.934133408Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1438-37ADBEA0.v1, size=374, entryCompress=NONE, stateCompress=NONE} 04:03:04,936 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:04,936 PM [leader758-%d210633] [DEBUG] RaftStorageCompleter - leader758: 1 more completion(s) pending 04:03:04,936 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 141.6 μs 04:03:04,936 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:04,937 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 7, leaderLastIndex: 6, leaderSnapShotIndex: 2 04:03:04,937 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader758: running 2 completion(s) 04:03:04,937 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader758: snapshot is durable as of 2025-08-11T16:03:04.934133408Z 04:03:04,937 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - leader758: Snapshot success, discarding journal entries up to 3 04:03:04,937 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 334.6 μs 04:03:04,937 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=4, applyTo=7] 04:03:04,937 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: discarded entries up to (and including) 3 04:03:04,937 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:04,937 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 216.8 μs 04:03:04,937 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:04,937 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower759 (Follower): Commit index set to 6 04:03:04,937 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 6 (of expected -1) 04:03:04,937 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower759 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 6, context.getLastApplied(): 2, lastIndex(): 6 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower759 (Follower): Setting last applied to 3 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower759: Moving last applied index from 2 to 3 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower759: Applying state for log index 3 data three 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower759: applyState called: three 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower759 (Follower): Setting last applied to 4 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower759: Moving last applied index from 3 to 4 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower759: Applying state for log index 4 data four 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower759: applyState called: four 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower759 (Follower): Setting last applied to 5 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower759: Moving last applied index from 4 to 5 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower759: Applying state for log index 5 data five 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower759: applyState called: five 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower759 (Follower): Setting last applied to 6 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower759: Moving last applied index from 5 to 6 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower759: Applying state for log index 6 data six 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower759: applyState called: six 04:03:04,938 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 1.683 ms 04:03:04,938 PM [follower759-writer-141] [DEBUG] EntryJournalV1 - follower759: updated meta to JournalMeta[replayFrom=1, applyTo=7] 04:03:04,938 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:04,938 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 158.1 μs 04:03:04,938 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:04,938 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower759: running 1 completion(s) 04:03:04,938 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:04,938 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorSnapshotMessageSupport - leader758: SnapshotComplete received [WARN] [08/11/2025 16:03:04.973] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:05,29 PM [awaitility-thread] [DEBUG] RaftStorage - leader758: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1438-37ADBEA0.v1 04:03:05,29 PM [awaitility-thread] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/TermInfo.properties 04:03:05,29 PM [awaitility-thread] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/journal-v1-1.log 04:03:05,29 PM [awaitility-thread] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/metadata-v1-1.log 04:03:05,29 PM [awaitility-thread] [DEBUG] RaftStorage - leader758: picked SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:04.934133408Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1438-37ADBEA0.v1, size=374, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:05,30 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderInstallsSnapshotWithRestartedFollowerDuringSnapshotInstallation: sending 1 more payload to trigger second snapshot 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader758: Persist data index=7 term=6 command=seven 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader758: Append log entry and persist JournaledLogEntry{index=7, term=6, command=seven, pending=true} 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Replicate message: identifier: MockIdentifier{value=seven}, logIndex: 7, isSendImmediate: true 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 7, leaderLastIndex: 7, leaderSnapShotIndex: 2 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): sendAppendEntries: 7 is present for follower follower759 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Sending AppendEntries to follower follower759: AppendEntries{term=6, leaderId=leader758, prevLogIndex=6, prevLogTerm=6, leaderCommit=6, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=7, term=6, command=seven, pending=true}]} 04:03:05,30 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 92.95 ms 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower760: active: false, followerNextIndex: 2, leaderLastIndex: 7, leaderSnapShotIndex: 2 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower759 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader758, prevLogIndex=6, prevLogTerm=6, leaderCommit=6, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=7, term=6, command=seven, pending=true}]} 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower759 (Follower): Number of entries to be appended = 1 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower759 (Follower): After cleanup, lastIndex: 6, entries to be added from: 0 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower759 (Follower): Append entry to log seven 04:03:05,30 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower759: Append log entry and persist JournaledLogEntry{index=7, term=6, command=seven, pending=true} 04:03:05,31 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower759 (Follower): Log size is now 6 04:03:05,31 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower759 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower759, logLastIndex=7, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:05,31 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 6 (of expected -1) 04:03:05,31 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 92.44 ms 04:03:05,31 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower759 updated: matchIndex: 7, nextIndex: 8 04:03:05,31 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower759: awaiting more completions to resolve 1 synchronized callback(s) 04:03:05,31 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): Checking sendAppendEntries for follower follower759: active: true, followerNextIndex: 8, leaderLastIndex: 7, leaderSnapShotIndex: 2 [WARN] [08/11/2025 16:03:05.031] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:05,31 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:05,31 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 1.104 ms 04:03:05,31 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:05,31 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:05,31 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader758: persist complete JournaledLogEntry{index=7, term=6, command=seven, pending=true} 04:03:05,31 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader758: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=6, lastAppliedTerm=6, lastIndex=7, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=1, mandatoryTrim=false] 04:03:05,31 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader758: captured snapshot at lastSequenceNumber: 7 04:03:05,31 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader758: Persising snapshot at EntryInfo[index=6, term=6]/EntryInfo[index=7, term=6] 04:03:05,31 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader758: support() called 04:03:05,32 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:05,32 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower759: running 1 completion(s) 04:03:05,32 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 993.9 μs 04:03:05,32 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:05,32 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower759: persist complete JournaledLogEntry{index=7, term=6, command=seven, pending=false} 04:03:05,32 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower759: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=7, term=6, command=seven, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@687dfe0f} 04:03:05,32 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower759: synchronized callbacks after 1.098 ms 04:03:05,32 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader758: log size 5 exceeds the snapshot batch count 4 - doing snapshotPreCommit with index 6 04:03:05,32 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader758: Removed in-memory snapshotted entries, adjusted snapshotIndex: 6 and term: 6 04:03:05,32 PM [leader758-%d210634] [DEBUG] RaftStorage - leader758: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-01EDA750.tmp 04:03:05,32 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader758 (Leader): Applying to log - commitIndex: 7, lastAppliedIndex: 6 04:03:05,32 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader758 (Leader): Setting last applied to 7 04:03:05,32 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader758: Moving last applied index from 6 to 7 04:03:05,32 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader758: Applying state for log index 7 data seven 04:03:05,32 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader758: applyState called: seven 04:03:05,32 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 901.0 μs 04:03:05,32 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=4, applyTo=8] 04:03:05,32 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:05,32 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 182.1 μs 04:03:05,32 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:05,32 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:05,35 PM [leader758-%d210634] [DEBUG] RaftStorage - leader758: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-01EDA750.v1 04:03:05,35 PM [leader758-%d210634] [DEBUG] RaftStorage - leader758: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1438-37ADBEA0.v1 04:03:05,35 PM [leader758-%d210634] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/TermInfo.properties 04:03:05,35 PM [leader758-%d210634] [DEBUG] RaftStorage - leader758: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-01EDA750.v1 04:03:05,35 PM [leader758-%d210634] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/journal-v1-1.log 04:03:05,35 PM [leader758-%d210634] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/metadata-v1-1.log 04:03:05,35 PM [leader758-%d210634] [DEBUG] RaftStorage - leader758: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:04.934133408Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1438-37ADBEA0.v1, size=374, entryCompress=NONE, stateCompress=NONE} 04:03:05,35 PM [leader758-%d210634] [DEBUG] RaftStorage - leader758: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-08-11T16:03:05.032352080Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-01EDA750.v1, size=442, entryCompress=NONE, stateCompress=NONE} 04:03:05,35 PM [leader758-%d210634] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:05,35 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:05,35 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader758: snapshot is durable as of 2025-08-11T16:03:05.032352080Z 04:03:05,35 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader758: Snapshot success, discarding journal entries up to 7 04:03:05,35 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorSnapshotMessageSupport - leader758: SnapshotComplete received 04:03:05,36 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 3.126 ms 04:03:05,36 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=8, applyTo=8] 04:03:05,36 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: discarded entries up to (and including) 7 04:03:05,36 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:05,36 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 213.1 μs 04:03:05,36 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:05,36 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) [WARN] [08/11/2025 16:03:05.094] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:05,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower759 (Follower): Commit index set to 7 04:03:05,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:05,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower759 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 7, context.getLastApplied(): 6, lastIndex(): 7 04:03:05,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower759 (Follower): Setting last applied to 7 04:03:05,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower759: Moving last applied index from 6 to 7 04:03:05,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower759: Applying state for log index 7 data seven 04:03:05,213 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower759: applyState called: seven [WARN] [08/11/2025 16:03:05.213] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower760] received dead letter from TestActor[pekko://test/user/leader758]: AppendEntries{term=6, leaderId=leader758, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 04:03:05,214 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 182.2 ms 04:03:05,214 PM [follower759-writer-141] [DEBUG] EntryJournalV1 - follower759: updated meta to JournalMeta[replayFrom=1, applyTo=8] 04:03:05,214 PM [follower759-writer-141] [DEBUG] RaftStorageCompleter - follower759: 1 completion(s) pending 04:03:05,214 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 322.2 μs 04:03:05,214 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: waiting to receive commands 04:03:05,214 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower759: running 1 completion(s) 04:03:05,232 PM [awaitility-thread] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/TermInfo.properties 04:03:05,232 PM [awaitility-thread] [DEBUG] RaftStorage - leader758: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-01EDA750.v1 04:03:05,232 PM [awaitility-thread] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/journal-v1-1.log 04:03:05,232 PM [awaitility-thread] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/metadata-v1-1.log 04:03:05,232 PM [awaitility-thread] [DEBUG] RaftStorage - leader758: picked SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-08-11T16:03:05.032352080Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-01EDA750.v1, size=442, entryCompress=NONE, stateCompress=NONE} as the latest file [WARN] [08/11/2025 16:03:05.235] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower760]: org.opendaylight.controller.cluster.common.actor.Monitor@7c66d4d6 04:03:05,236 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower760: Starting recovery with journal batch size 1000 04:03:05,236 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: started executor 04:03:05,236 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: started executor 04:03:05,237 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower760: metadata journal open: replayFrom=1 applyTo=2 04:03:05,237 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower760: journal open: firstIndex=1 lastIndex=2 replayFrom=1 fileEntries=0 04:03:05,237 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - follower760: journal open: applyTo=2 04:03:05,237 PM [follower760-writer-144] [DEBUG] JournalWriteTask - follower760: journal writer started 04:03:05,238 PM [follower760-writer-144] [DEBUG] JournalWriteTask - follower760: waiting to receive commands 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower760: Set currentTerm=6, votedFor=leader758 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower760: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower760: restoring local TermInfo{term=6, votedFor=leader758} 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower760: Set currentTerm=6, votedFor=leader758 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower760: Pekko recovery completed and cannot restore from snapshot 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower760: starting journal recovery 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower760: Moving last applied index from -1 to -1 04:03:05,239 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower760: applying entries up to 2 04:03:05,240 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower760: recovered journal DefaultLogEntry[index=0, term=6, command=zero] 04:03:05,240 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower760: recovered journal DefaultLogEntry[index=1, term=6, command=one] 04:03:05,240 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower760: Moving last applied index from -1 to 1 04:03:05,240 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower760: journal recovery completedin 1.001 ms with journalIndex=1 04:03:05,240 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower760: Journal recovery completed and cannot restore from snapshot 04:03:05,334 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:05,334 PM [test-pekko.actor.default-dispatcher-17] [WARN] AbstractLeader - leader758 (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply{term=6, success=true, followerId=follower760, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, timeSinceLastActivity : 840, lastApplied : 7, commitIndex : 7 04:03:05,334 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractLeader - leader758 (Leader): Initiating install snapshot to follower follower760: follower nextIndex: 2, leader snapshotIndex: 6, leader lastIndex: 7, leader log size: 1 04:03:05,334 PM [test-pekko.actor.default-dispatcher-17] [INFO] SnapshotManager - leader758: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=7, lastAppliedTerm=6, lastIndex=7, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower760 04:03:05,334 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SnapshotManager - leader758: last applied journal index prior to capture: 8 04:03:05,334 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader758: support() called 04:03:05,335 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower760: Last sync leader does not match current leader leader758, need to catch up to -1 04:03:05,335 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower760: fakeSnapshot purging log to 0 for term 6 04:03:05,335 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower760: Lagging -2 entries behind leader leader758 and reached 1 (of expected -1) 04:03:05,335 PM [leader758-%d210635] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:05,335 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) [WARN] [08/11/2025 16:03:05.335] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower760] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower760, initialSyncDone=false] 04:03:05,335 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader758: Persising snapshot at EntryInfo[index=7, term=6]/EntryInfo[index=7, term=6] 04:03:05,335 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader758: support() called [WARN] [08/11/2025 16:03:05.335] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower760] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower760, initialSyncDone=true] 04:03:05,336 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader758: Removed in-memory snapshotted entries, adjusted snapshotIndex: 6 and term: 6 04:03:05,336 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader758 (Leader): sendInstallSnapshot 04:03:05,336 PM [leader758-%d210636] [DEBUG] RaftStorage - leader758: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-1409090B.tmp 04:03:05,336 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] LeaderInstallSnapshotState - leader758 (Leader): Snapshot 396 bytes, total chunks to send: 1 04:03:05,336 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] LeaderInstallSnapshotState - leader758 (Leader): Next chunk: total length=396, offset=0, size=396, hashCode=-153134944 04:03:05,336 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader758 (Leader): next snapshot chunk size for follower follower760: 396 04:03:05,336 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader758 (Leader): InstallSnapshot sent to follower Vector(user, follower760), Chunk: 1/1 04:03:05,336 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower760 (Follower): handleInstallSnapshot: InstallSnapshot{term=6, leaderId=leader758, lastIncludedIndex=7, lastIncludedTerm=6, datasize=396, chunk=1/1, lastChunkHashCode=OptionalInt[-1]} 04:03:05,336 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower760: Lagging 6 entries behind leader leader758 and reached 1 (of expected -1) 04:03:05,336 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotTracker - follower760 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 04:03:05,336 PM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower760 (Follower): Snapshot received from leader: leader758 04:03:05,336 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - follower760: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=396}} 04:03:05,336 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower760: support() called 04:03:05,337 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower760: Converted InstallSnapshot from leader: leader758 to state 04:03:05,337 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower760: last applied journal index prior to persisting applied snapshot: 2 04:03:05,337 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower760: support() called 04:03:05,338 PM [follower760-%d298647] [DEBUG] RaftStorage - follower760: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/snapshot-00000000689A1439-14251630.tmp 04:03:05,339 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EnabledRaftStorage - follower760: terminating thread VirtualThread[#936,follower760-writer-144]/waiting 04:03:05,339 PM [follower760-writer-144] [DEBUG] JournalWriteTask - follower760: received 1 commands after 101.9 ms 04:03:05,340 PM [follower760-writer-144] [DEBUG] JournalWriteTask - follower760: completed 1 commands in 16.96 μs 04:03:05,340 PM [follower760-writer-144] [DEBUG] JournalWriteTask - follower760: journal writer stopped 04:03:05,340 PM [leader758-%d210636] [DEBUG] RaftStorage - leader758: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-1409090B.v1 04:03:05,340 PM [leader758-%d210636] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/TermInfo.properties 04:03:05,340 PM [leader758-%d210636] [DEBUG] RaftStorage - leader758: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-01EDA750.v1 04:03:05,340 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower760: journal closed 04:03:05,340 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - follower760: journal closed 04:03:05,340 PM [leader758-%d210636] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/journal-v1-1.log 04:03:05,340 PM [leader758-%d210636] [DEBUG] RaftStorage - leader758: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-1409090B.v1 04:03:05,340 PM [leader758-%d210636] [DEBUG] RaftStorage - leader758: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/metadata-v1-1.log 04:03:05,340 PM [leader758-%d210636] [DEBUG] RaftStorage - leader758: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=6, term=6], timestamp=2025-08-11T16:03:05.032352080Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-01EDA750.v1, size=442, entryCompress=NONE, stateCompress=NONE} 04:03:05,340 PM [leader758-%d210636] [DEBUG] RaftStorage - leader758: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=7, term=6], timestamp=2025-08-11T16:03:05.336136459Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/leader758/snapshot-00000000689A1439-1409090B.v1, size=460, entryCompress=NONE, stateCompress=NONE} 04:03:05,340 PM [leader758-%d210636] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:05,341 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:05,341 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader758: snapshot is durable as of 2025-08-11T16:03:05.336136459Z 04:03:05,341 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - leader758: Snapshot success, discarding journal entries up to 8 04:03:05,341 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorSnapshotMessageSupport - leader758: SnapshotComplete received 04:03:05,341 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: stopped executor with 0 remaining tasks 04:03:05,341 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 305.6 ms 04:03:05,343 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: stopped executor with 0 remaining tasks 04:03:05,343 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: updated meta to JournalMeta[replayFrom=9, applyTo=8] 04:03:05,343 PM [leader758-writer-143] [DEBUG] EntryJournalV1 - leader758: discarded entries up to (and including) 8 04:03:05,343 PM [leader758-writer-143] [DEBUG] RaftStorageCompleter - leader758: 1 completion(s) pending 04:03:05,343 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader758: running 1 completion(s) 04:03:05,343 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 323.6 μs 04:03:05,343 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: waiting to receive commands 04:03:05,345 PM [follower760-%d298647] [DEBUG] RaftStorageCompleter - follower760: 1 completion(s) pending [WARN] [08/11/2025 16:03:05.347] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower760] received dead letter: ExecuteInSelfMessage[runnable=org.opendaylight.controller.cluster.raft.spi.RaftStorageCompleter$$Lambda/0x00000007c0438428@38bba766] [WARN] [08/11/2025 16:03:05.443] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower760]: org.opendaylight.controller.cluster.common.actor.Monitor@5f8f9bd9 04:03:05,444 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower760: Starting recovery with journal batch size 1000 04:03:05,444 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: started executor 04:03:05,444 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: started executor 04:03:05,445 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower760: metadata journal open: replayFrom=1 applyTo=2 04:03:05,446 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower760: journal open: firstIndex=1 lastIndex=2 replayFrom=1 fileEntries=0 04:03:05,446 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - follower760: journal open: applyTo=2 04:03:05,446 PM [follower760-writer-145] [DEBUG] JournalWriteTask - follower760: journal writer started 04:03:05,446 PM [follower760-writer-145] [DEBUG] JournalWriteTask - follower760: waiting to receive commands 04:03:05,447 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower760: Set currentTerm=6, votedFor=leader758 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower760: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower760: restoring local TermInfo{term=6, votedFor=leader758} 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower760: Set currentTerm=6, votedFor=leader758 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower760: Pekko recovery completed and cannot restore from snapshot 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower760: starting journal recovery 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower760: Moving last applied index from -1 to -1 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower760: applying entries up to 2 04:03:05,448 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower760: recovered journal DefaultLogEntry[index=0, term=6, command=zero] 04:03:05,449 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower760: recovered journal DefaultLogEntry[index=1, term=6, command=one] 04:03:05,450 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower760: Moving last applied index from -1 to 1 04:03:05,450 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower760: journal recovery completedin 1.656 ms with journalIndex=1 04:03:05,450 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower760: Journal recovery completed and cannot restore from snapshot 04:03:05,453 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:05,547 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:05,547 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:05,547 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:05,547 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:05,574 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:05,648 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:05,648 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:05,648 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:05,648 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:05,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:05,749 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:05,749 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:05,749 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:05,749 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:05,814 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:05,814 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower760: Last sync leader does not match current leader leader758, need to catch up to -1 04:03:05,814 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SnapshotManager - follower760: fakeSnapshot purging log to 0 for term 6 04:03:05,814 PM [test-pekko.actor.default-dispatcher-6] [WARN] AbstractLeader - leader758 (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply{term=6, success=true, followerId=follower760, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, timeSinceLastActivity : 478, lastApplied : 7, commitIndex : 7 [WARN] [08/11/2025 16:03:05.814] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower760] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower760, initialSyncDone=false] 04:03:05,850 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:05,850 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:05,850 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:05,850 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:05,934 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:05,950 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:05,951 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:05,951 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:05,951 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:06,51 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:06,51 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:06,51 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:06,51 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:06,53 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:06,152 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:06,152 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:06,152 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:06,152 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:06,174 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:06,253 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:06,253 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:06,253 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:06,253 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:06,294 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower760: Lagging -2 entries behind leader leader758 and reached 1 (of expected -1) 04:03:06,294 PM [test-pekko.actor.default-dispatcher-16] [WARN] AbstractLeader - leader758 (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply{term=6, success=true, followerId=follower760, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, timeSinceLastActivity : 479, lastApplied : 7, commitIndex : 7 [WARN] [08/11/2025 16:03:06.294] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower760] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower760, initialSyncDone=true] 04:03:06,294 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:06,354 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:06,354 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:06,354 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:06,354 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:06,414 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:06,455 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:06,455 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:06,455 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:06,455 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:06,534 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:06,555 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:06,555 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:06,555 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:06,555 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:06,654 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:06,656 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:06,656 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:06,656 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:06,656 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:06,756 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:06,756 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:06,756 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:06,756 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:06,774 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower760: Lagging -2 entries behind leader leader758 and reached 1 (of expected -1) 04:03:06,774 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:06,774 PM [test-pekko.actor.default-dispatcher-11] [WARN] AbstractLeader - leader758 (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply{term=6, success=true, followerId=follower760, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, timeSinceLastActivity : 479, lastApplied : 7, commitIndex : 7 04:03:06,857 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:06,857 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:06,857 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:06,857 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:06,894 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:06,957 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:06,958 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:06,958 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:06,958 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:07,14 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:07,58 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:07,58 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:07,58 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:07,58 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:07,134 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:07,159 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:07,159 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:07,159 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:07,159 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:07,254 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:07,254 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower760: Lagging -2 entries behind leader leader758 and reached 1 (of expected -1) 04:03:07,254 PM [test-pekko.actor.default-dispatcher-17] [WARN] AbstractLeader - leader758 (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply{term=6, success=true, followerId=follower760, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, timeSinceLastActivity : 480, lastApplied : 7, commitIndex : 7 04:03:07,260 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:07,260 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:07,260 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:07,260 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:07,361 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:07,361 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:07,361 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:07,361 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: no eligible files found 04:03:07,373 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): next snapshot chunk size for follower follower760: 396 04:03:07,373 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower759: Lagging 0 entries behind leader leader758 and reached 7 (of expected -1) 04:03:07,373 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader758 (Leader): InstallSnapshot sent to follower Vector(user, follower760), Chunk: 1/1 04:03:07,373 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower760 (Follower): handleInstallSnapshot: InstallSnapshot{term=6, leaderId=leader758, lastIncludedIndex=7, lastIncludedTerm=6, datasize=396, chunk=1/1, lastChunkHashCode=OptionalInt[-1]} 04:03:07,373 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower760: Lagging 6 entries behind leader leader758 and reached 1 (of expected -1) 04:03:07,373 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotTracker - follower760 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 04:03:07,373 PM [test-pekko.actor.default-dispatcher-7] [INFO] Follower - follower760 (Follower): Snapshot received from leader: leader758 04:03:07,374 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - follower760: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=396}} 04:03:07,374 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower760: support() called 04:03:07,375 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower760: Converted InstallSnapshot from leader: leader758 to state 04:03:07,375 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower760: last applied journal index prior to persisting applied snapshot: 2 04:03:07,375 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower760: support() called 04:03:07,375 PM [follower760-%d466162] [DEBUG] RaftStorage - follower760: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/snapshot-00000000689A143B-165F34B5.tmp 04:03:07,380 PM [follower760-%d466162] [DEBUG] RaftStorage - follower760: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/snapshot-00000000689A143B-165F34B5.v1 04:03:07,380 PM [follower760-%d466162] [DEBUG] RaftStorage - follower760: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/snapshot-00000000689A143B-165F34B5.v1 04:03:07,380 PM [follower760-%d466162] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:07,380 PM [follower760-%d466162] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:07,380 PM [follower760-%d466162] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:07,380 PM [follower760-%d466162] [DEBUG] RaftStorage - follower760: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=7, term=6], timestamp=2025-08-11T16:03:07.375338165Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/snapshot-00000000689A143B-165F34B5.v1, size=460, entryCompress=NONE, stateCompress=NONE} 04:03:07,380 PM [follower760-%d466162] [DEBUG] RaftStorageCompleter - follower760: 1 completion(s) pending 04:03:07,380 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower760: running 1 completion(s) 04:03:07,380 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - follower760: snapshot is durable as of 2025-08-11T16:03:07.375338165Z 04:03:07,381 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower760: support() called 04:03:07,381 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower760: applySnapshot called 04:03:07,381 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower760 (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=6, followerId=follower760, chunkIndex=1, success=true} 04:03:07,381 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower760: Snapshot success, discarding journal entries up to 2 04:03:07,381 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorSnapshotMessageSupport - follower760: SnapshotComplete received 04:03:07,381 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader758 (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=6, followerId=follower760, chunkIndex=1, success=true} 04:03:07,381 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractLeader - leader758 (Leader): Snapshot successfully installed on follower follower760 (last chunk 1) - matchIndex set to 7, nextIndex set to 8 04:03:07,381 PM [follower760-writer-145] [DEBUG] JournalWriteTask - follower760: received 1 commands after 1.935 s 04:03:07,381 PM [follower760-writer-145] [DEBUG] EntryJournalV1 - follower760: updated meta to JournalMeta[replayFrom=3, applyTo=2] 04:03:07,381 PM [follower760-writer-145] [DEBUG] EntryJournalV1 - follower760: discarded entries up to (and including) 2 04:03:07,381 PM [follower760-writer-145] [DEBUG] RaftStorageCompleter - follower760: 1 completion(s) pending 04:03:07,382 PM [follower760-writer-145] [DEBUG] JournalWriteTask - follower760: completed 1 commands in 287.2 μs 04:03:07,382 PM [follower760-writer-145] [DEBUG] JournalWriteTask - follower760: waiting to receive commands 04:03:07,382 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower760: running 1 completion(s) 04:03:07,461 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/snapshot-00000000689A143B-165F34B5.v1 04:03:07,461 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/TermInfo.properties 04:03:07,462 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/journal-v1-1.log 04:03:07,462 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/metadata-v1-1.log 04:03:07,462 PM [awaitility-thread] [DEBUG] RaftStorage - follower760: picked SnapshotFileV1{lastIncluded=EntryInfo[index=7, term=6], timestamp=2025-08-11T16:03:07.375338165Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16857394262091666550/follower760/snapshot-00000000689A143B-165F34B5.v1, size=460, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:07,462 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower759-collector761#213551166] 04:03:07,463 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower759] 04:03:07,464 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EnabledRaftStorage - follower759: terminating thread VirtualThread[#928,follower759-writer-141]/waiting 04:03:07,464 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: received 1 commands after 2.250 s 04:03:07,464 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: completed 1 commands in 14.34 μs 04:03:07,464 PM [follower759-writer-141] [DEBUG] JournalWriteTask - follower759: journal writer stopped 04:03:07,464 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower759: journal closed 04:03:07,464 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - follower759: journal closed 04:03:07,464 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower759: stopped executor with 0 remaining tasks 04:03:07,465 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower759: stopped executor with 0 remaining tasks 04:03:07,465 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower760-collector762#1712403765] 04:03:07,465 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower760] 04:03:07,465 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader758-collector763#986769251] 04:03:07,465 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader758] 04:03:07,466 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EnabledRaftStorage - leader758: terminating thread VirtualThread[#930,leader758-writer-143]/waiting 04:03:07,466 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: received 1 commands after 2.123 s 04:03:07,466 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: completed 1 commands in 12.47 μs 04:03:07,466 PM [leader758-writer-143] [DEBUG] JournalWriteTask - leader758: journal writer stopped 04:03:07,466 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader758: journal closed 04:03:07,466 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - leader758: journal closed 04:03:07,466 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader758: stopped executor with 0 remaining tasks 04:03:07,466 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader758: stopped executor with 0 remaining tasks 04:03:07,466 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower760-collector764#-1123719470] 04:03:07,467 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower760] 04:03:07,467 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower760-collector765#-2076003096] 04:03:07,467 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower760] 04:03:07,467 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EnabledRaftStorage - follower760: terminating thread VirtualThread[#940,follower760-writer-145]/waiting 04:03:07,467 PM [follower760-writer-145] [DEBUG] JournalWriteTask - follower760: received 1 commands after 85.48 ms 04:03:07,467 PM [follower760-writer-145] [DEBUG] JournalWriteTask - follower760: completed 1 commands in 12.83 μs 04:03:07,467 PM [follower760-writer-145] [DEBUG] JournalWriteTask - follower760: journal writer stopped 04:03:07,467 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower760: journal closed 04:03:07,467 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - follower760: journal closed 04:03:07,467 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: stopped executor with 0 remaining tasks 04:03:07,467 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower760: stopped executor with 0 remaining tasks 04:03:07,470 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotTriggeredByMemoryThresholdExceededWithLaggingFollower starting [WARN] [08/11/2025 16:03:07.473] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower770]: org.opendaylight.controller.cluster.common.actor.Monitor@3d00f988 04:03:07,474 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower770: Starting recovery with journal batch size 1000 04:03:07,474 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower770: started executor 04:03:07,474 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower770: started executor 04:03:07,474 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower770: metadata journal open: replayFrom=1 applyTo=0 04:03:07,475 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower770: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:07,475 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - follower770: journal open: applyTo=0 04:03:07,475 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: journal writer started 04:03:07,475 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:07,476 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower770/TermInfo.properties does not exist 04:03:07,476 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower770: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower770/journal-v1-1.log 04:03:07,476 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower770: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower770/metadata-v1-1.log 04:03:07,476 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower770: no eligible files found [WARN] [08/11/2025 16:03:07.476] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower771]: org.opendaylight.controller.cluster.common.actor.Monitor@7fac4150 04:03:07,476 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower770: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:07,477 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower771: Starting recovery with journal batch size 1000 04:03:07,477 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower771: started executor 04:03:07,477 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower771: started executor 04:03:07,478 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower771: metadata journal open: replayFrom=1 applyTo=0 04:03:07,478 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower771: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:07,478 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - follower771: journal open: applyTo=0 04:03:07,479 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: journal writer started 04:03:07,479 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:07,480 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower771/TermInfo.properties does not exist 04:03:07,480 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower771/journal-v1-1.log 04:03:07,480 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower771/metadata-v1-1.log 04:03:07,480 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower771: no eligible files found 04:03:07,480 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower770: Set currentTerm=0, votedFor=null 04:03:07,480 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower771: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:07,480 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower770: Local TermInfo store seeded with TermInfo{term=0} 04:03:07,480 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower770: Pekko recovery completed and can restore from snapshot 04:03:07,480 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower770: starting journal recovery 04:03:07,480 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from -1 to -1 04:03:07,480 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower770: empty journal: appending 0 entries 04:03:07,480 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:07,480 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower770: journal recovery completedin 277.4 μs with journalIndex=1 04:03:07,480 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower770: Journal recovery completed and can restore from snapshot 04:03:07,483 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower771: Set currentTerm=0, votedFor=null 04:03:07,483 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower771: Local TermInfo store seeded with TermInfo{term=0} 04:03:07,483 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower771: Pekko recovery completed and can restore from snapshot 04:03:07,483 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower771: starting journal recovery 04:03:07,483 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower771: Moving last applied index from -1 to -1 04:03:07,483 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower771: empty journal: appending 0 entries 04:03:07,483 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:07,483 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower771: journal recovery completedin 169.5 μs with journalIndex=1 04:03:07,483 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower771: Journal recovery completed and can restore from snapshot [WARN] [08/11/2025 16:03:07.578] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader769]: org.opendaylight.controller.cluster.common.actor.Monitor@68837178 04:03:07,579 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - leader769: Starting recovery with journal batch size 1000 04:03:07,579 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: started executor 04:03:07,579 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: started executor 04:03:07,580 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader769: metadata journal open: replayFrom=1 applyTo=0 04:03:07,580 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader769: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 04:03:07,580 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - leader769: journal open: applyTo=0 04:03:07,581 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: journal writer started 04:03:07,581 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:07,582 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties does not exist 04:03:07,582 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:07,582 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:07,582 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: no eligible files found 04:03:07,582 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=5, votedFor=leader769 04:03:07,582 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader769: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 04:03:07,588 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=5, votedFor=leader769 04:03:07,588 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader769: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader769} 04:03:07,588 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader769: taking snapshot to clear Pekko persistence to 1 04:03:07,588 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Recovery - leader769: no applied entries in recovery log, re-snapshotting EntryInfo[index=-1, term=-1] 04:03:07,588 PM [test-pekko.actor.default-dispatcher-6] [INFO] Recovery - leader769: Taking snapshot on entry with index -1 04:03:07,588 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader769: support() called 04:03:07,588 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143B-23162629.tmp 04:03:07,590 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143B-23162629.v1 04:03:07,590 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143B-23162629.v1 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:07.588654121Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143B-23162629.v1, size=221, entryCompress=NONE, stateCompress=NONE} 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [INFO] Recovery - leader769: Snapshot completed in 3.078 ms, resetting timer for the next recovery snapshot 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: Pekko recovery completed and cannot restore from snapshot 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: starting journal recovery 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from -1 to -1 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader769: empty journal: appending 0 entries 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=0] 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader769: journal recovery completedin 173.6 μs with journalIndex=1 04:03:07,591 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: Journal recovery completed and cannot restore from snapshot 04:03:07,592 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: ignoring DeleteMessagesSuccess(1) 04:03:08,23 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 04:03:08,23 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - leader769 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.10:17355, Removed) 04:03:08,23 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - leader769 (Follower): Received ElectionTimeout - switching to Candidate 04:03:08,23 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader769 (Candidate): Election: Candidate has following voting peers: [follower771, follower770] 04:03:08,27 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=6, votedFor=leader769 04:03:08,27 PM [test-pekko.actor.default-dispatcher-11] [INFO] Candidate - leader769 (Candidate): Starting new election term 6 04:03:08,27 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader769 (Candidate): Sending RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} to peer follower771 04:03:08,27 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader769 (Candidate): Sending RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} to peer follower770 04:03:08,27 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower771 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 04:03:08,27 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower770 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 04:03:08,27 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader769 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 04:03:08,27 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower770 (Follower): Cluster state: [] 04:03:08,27 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower770 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 04:03:08,27 PM [test-pekko.actor.default-dispatcher-7] [INFO] Follower - follower770 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 04:03:08,27 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower771 (Follower): Cluster state: [] 04:03:08,27 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower771 (Follower): Candidate in requestVote:RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 04:03:08,27 PM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower771 (Follower): Term 6 in "RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 04:03:08,30 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower770: Set currentTerm=6, votedFor=null 04:03:08,30 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower770 (Follower): In requestVote: RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 04:03:08,32 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower771: Set currentTerm=6, votedFor=null 04:03:08,32 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower771 (Follower): In requestVote: RequestVote{term=6, candidateId=leader769, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 6, votedFor: null, lastIndex: -1, lastTerm: -1 04:03:08,33 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower770: Set currentTerm=6, votedFor=leader769 04:03:08,33 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower770 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 04:03:08,33 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - leader769 (Candidate): RaftRPC message received RequestVoteReply{term=6, voteGranted=true}, my term is 6 04:03:08,33 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - leader769 (Candidate): handleRequestVoteReply: RequestVoteReply{term=6, voteGranted=true}, current voteCount: 1 04:03:08,33 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Election: Leader has following peers: [follower771, follower770] 04:03:08,33 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:08,33 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 04:03:08,33 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActorBehavior - leader769 (Candidate) :- Switching from behavior Candidate to Leader, election term: 6 04:03:08,33 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Last sync leader does not match current leader leader769, need to catch up to -1 04:03:08,34 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: -1, nextIndex: 0 04:03:08,34 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/11/2025 16:03:08.034] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower770] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower770, initialSyncDone=false] 04:03:08,34 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower771: Set currentTerm=6, votedFor=leader769 04:03:08,34 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower771 (Follower): requestVote returning: RequestVoteReply{term=6, voteGranted=true} 04:03:08,34 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower771: Last sync leader does not match current leader leader769, need to catch up to -1 04:03:08,34 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower771 updated: matchIndex: -1, nextIndex: 0 04:03:08,34 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [08/11/2025 16:03:08.034] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower771] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower771, initialSyncDone=false] 04:03:08,45 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - Leader created and elected 04:03:08,45 PM [main] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143B-23162629.v1 04:03:08,45 PM [main] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties 04:03:08,45 PM [main] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:08,45 PM [main] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:08,45 PM [main] [DEBUG] RaftStorage - leader769: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:07.588654121Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143B-23162629.v1, size=221, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:08,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader769: Persist data index=0 term=6 command=zero 04:03:08,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:08,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 04:03:08,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:08,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 0 is present for follower follower771 04:03:08,46 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 465.4 ms 04:03:08,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower771: AppendEntries{term=6, leaderId=leader769, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:08,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:08,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 0 is present for follower follower770 04:03:08,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:08,46 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:08,46 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): Number of entries to be appended = 1 04:03:08,46 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 04:03:08,46 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): Append entry to log zero 04:03:08,46 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower771: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:08,46 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=6, command=zero, pending=true}]} 04:03:08,46 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 567.9 ms 04:03:08,46 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): Log size is now 1 04:03:08,47 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower771, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:08,47 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached -1 (of expected -1) 04:03:08,47 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower771: awaiting more completions to resolve 1 synchronized callback(s) 04:03:08,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 04:03:08,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 04:03:08,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower770 (Follower): Append entry to log zero 04:03:08,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:08,47 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower771 updated: matchIndex: 0, nextIndex: 1 04:03:08,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower770 (Follower): Log size is now 1 04:03:08,47 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:08,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:08,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached -1 (of expected -1) 04:03:08,47 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 0, nextIndex: 1 04:03:08,47 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 04:03:08,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower770: awaiting more completions to resolve 1 synchronized callback(s) 04:03:08,47 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 572.3 ms 04:03:08,51 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:08,52 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 5.414 ms 04:03:08,52 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:08,52 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:08,52 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=true} 04:03:08,52 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 04:03:08,52 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 0 04:03:08,52 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from -1 to 0 04:03:08,52 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: Applying state for log index 0 data zero 04:03:08,52 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader769: applyState called: zero 04:03:08,52 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 589.4 μs 04:03:08,52 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=1] 04:03:08,52 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:08,52 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 171.7 μs 04:03:08,52 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:08,52 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:08,53 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:08,53 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 5.590 ms 04:03:08,53 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:08,53 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:08,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:08,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower771: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:03:08,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower771: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=6, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@4237fb6b} 04:03:08,53 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 6.412 ms 04:03:08,53 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:08,53 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower771: synchronized callbacks after 6.357 ms 04:03:08,53 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:08,53 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=0, term=6, command=zero, pending=false} 04:03:08,53 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower770: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=6, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@2a4ebea4} [WARN] [08/11/2025 16:03:08.053] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower771] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower771, initialSyncDone=true] 04:03:08,53 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower770: synchronized callbacks after 6.080 ms [WARN] [08/11/2025 16:03:08.054] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower770] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower770, initialSyncDone=true] 04:03:08,154 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): Commit index set to 0 04:03:08,154 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 04:03:08,154 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 04:03:08,154 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower771 (Follower): Setting last applied to 0 04:03:08,154 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower771: Moving last applied index from -1 to 0 04:03:08,154 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower771: Applying state for log index 0 data zero 04:03:08,154 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower771: applyState called: zero 04:03:08,154 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): Commit index set to 0 04:03:08,154 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 100.9 ms 04:03:08,154 PM [follower771-writer-147] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=1, applyTo=1] 04:03:08,154 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:08,154 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:08,154 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 359.8 μs 04:03:08,155 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:08,155 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 04:03:08,155 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 04:03:08,155 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 0 04:03:08,155 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from -1 to 0 04:03:08,155 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower770: Applying state for log index 0 data zero 04:03:08,155 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower770: applyState called: zero 04:03:08,155 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 102.0 ms 04:03:08,155 PM [follower770-writer-146] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=1] 04:03:08,155 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:08,155 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 172.6 μs 04:03:08,155 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:08,155 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:08,273 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 04:03:08,273 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 04:03:08,394 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 04:03:08,514 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 04:03:08,634 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 04:03:08,704 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Persist data index=1 term=6 command=one 04:03:08,704 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:03:08,704 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 04:03:08,704 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:08,704 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 652.1 ms 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 1 is present for follower follower770 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=0, prevLogTerm=6, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=6, command=one, pending=true}]} 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): Append entry to log one 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): Log size is now 2 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=1, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 0 (of expected -1) 04:03:08,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower770: awaiting more completions to resolve 1 synchronized callback(s) 04:03:08,705 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 550.0 ms 04:03:08,705 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 1, nextIndex: 2 04:03:08,706 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: -1 04:03:08,707 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:08,707 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 2.520 ms 04:03:08,707 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:08,707 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:08,707 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:08,707 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=true} 04:03:08,707 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 1.758 ms 04:03:08,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:08,707 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:08,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=1, term=6, command=one, pending=false} 04:03:08,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower770: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=6, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@60435551} 04:03:08,707 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 04:03:08,707 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower770: synchronized callbacks after 1.911 ms 04:03:08,707 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 1 04:03:08,707 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 0 to 1 04:03:08,707 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader769: Applying state for log index 1 data one 04:03:08,707 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader769: applyState called: one 04:03:08,708 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader769: fakeSnapshot purging log to 0 for term 6 04:03:08,708 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 589.8 μs 04:03:08,708 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:03:08,708 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:08,708 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 159.2 μs 04:03:08,708 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:08,708 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:08,873 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): Commit index set to 1 04:03:08,873 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:08,873 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 04:03:08,873 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 1 04:03:08,873 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 0 to 1 04:03:08,873 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower770: Applying state for log index 1 data one 04:03:08,873 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower770: applyState called: one 04:03:08,873 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower770: fakeSnapshot purging log to 0 for term 6 04:03:08,873 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 165.8 ms 04:03:08,873 PM [follower770-writer-146] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=2] 04:03:08,874 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:08,874 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 415.7 μs 04:03:08,874 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:08,874 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:08,994 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:09,114 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:09,234 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:09,353 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:09,463 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:09,584 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:09,704 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:09,813 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:09,933 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:10,43 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:10,163 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:10,284 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:10,393 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:10,514 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:10,634 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:10,754 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:10,755 PM [main] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143B-23162629.v1 04:03:10,755 PM [main] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties 04:03:10,755 PM [main] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:10,755 PM [main] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:10,755 PM [main] [DEBUG] RaftStorage - leader769: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:07.588654121Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143B-23162629.v1, size=221, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:10,874 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:10,993 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:11,114 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:11,161 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: Persist data index=2 term=6 command=two 04:03:11,161 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:11,161 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=two}, logIndex: 2, isSendImmediate: true 04:03:11,161 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: false, followerNextIndex: 1, leaderLastIndex: 2, leaderSnapShotIndex: 0 04:03:11,161 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 2.453 s 04:03:11,161 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 04:03:11,161 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 2 is present for follower follower770 04:03:11,161 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 04:03:11,162 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=1, prevLogTerm=6, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=6, command=two, pending=true}]} 04:03:11,162 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 04:03:11,162 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 04:03:11,162 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): Append entry to log two 04:03:11,162 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:11,162 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): Log size is now 2 04:03:11,162 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=2, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:11,162 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 1 (of expected -1) 04:03:11,162 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower770: awaiting more completions to resolve 1 synchronized callback(s) 04:03:11,162 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 2.288 s 04:03:11,162 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 2, nextIndex: 3 04:03:11,162 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 0 04:03:11,287 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:11,287 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:11,287 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:11,287 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 124.9 ms 04:03:11,287 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:11,287 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=false} 04:03:11,287 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 125.5 ms 04:03:11,287 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:11,287 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:11,287 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower770: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=6, command=two, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@4c5d28ee} 04:03:11,287 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower770: synchronized callbacks after 125.1 ms 04:03:11,287 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=2, term=6, command=two, pending=true} 04:03:11,287 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=1, lastAppliedTerm=6, lastIndex=2, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=1, mandatoryTrim=false] 04:03:11,287 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader769: captured snapshot at lastSequenceNumber: 2 04:03:11,287 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Persising snapshot at EntryInfo[index=1, term=6]/EntryInfo[index=2, term=6] 04:03:11,287 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: support() called 04:03:11,288 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader769: log data size 705 exceeds the memory threshold 700 - doing snapshotPreCommit with index 1 04:03:11,288 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Removed in-memory snapshotted entries, adjusted snapshotIndex: 1 and term: 6 04:03:11,288 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 04:03:11,288 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 2 04:03:11,288 PM [leader769-%d695320] [DEBUG] RaftStorage - leader769: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-112C4FDC.tmp 04:03:11,288 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 1 to 2 04:03:11,288 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader769: Applying state for log index 2 data two 04:03:11,288 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: applyState called: two 04:03:11,288 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 1.071 ms 04:03:11,288 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=3] 04:03:11,288 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:11,288 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 222.1 μs 04:03:11,288 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:11,288 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:11,291 PM [leader769-%d695320] [DEBUG] RaftStorage - leader769: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-112C4FDC.v1 04:03:11,291 PM [leader769-%d695320] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143B-23162629.v1 04:03:11,291 PM [leader769-%d695320] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-112C4FDC.v1 04:03:11,291 PM [leader769-%d695320] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties 04:03:11,291 PM [leader769-%d695320] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:11,291 PM [leader769-%d695320] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:11,292 PM [leader769-%d695320] [DEBUG] RaftStorage - leader769: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-08-11T16:03:07.588654121Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143B-23162629.v1, size=221, entryCompress=NONE, stateCompress=NONE} 04:03:11,292 PM [leader769-%d695320] [DEBUG] RaftStorage - leader769: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=1, term=6], timestamp=2025-08-11T16:03:11.288116700Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-112C4FDC.v1, size=358, entryCompress=NONE, stateCompress=NONE} 04:03:11,292 PM [leader769-%d695320] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:11,292 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:11,292 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - leader769: snapshot is durable as of 2025-08-11T16:03:11.288116700Z 04:03:11,292 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - leader769: Snapshot success, discarding journal entries up to 2 04:03:11,292 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorSnapshotMessageSupport - leader769: SnapshotComplete received 04:03:11,292 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 3.639 ms 04:03:11,292 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=3, applyTo=3] 04:03:11,292 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: discarded entries up to (and including) 2 04:03:11,292 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:11,292 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 182.3 μs 04:03:11,292 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:11,292 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:11,353 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): Commit index set to 2 04:03:11,354 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 04:03:11,354 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 2 04:03:11,354 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 2 04:03:11,354 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 1 to 2 04:03:11,354 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower770: Applying state for log index 2 data two 04:03:11,354 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower770: applyState called: two 04:03:11,354 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 67.14 ms 04:03:11,355 PM [follower770-writer-146] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=3] 04:03:11,355 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:11,355 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 492.0 μs 04:03:11,355 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:11,355 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:11,362 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-112C4FDC.v1 04:03:11,363 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties 04:03:11,363 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:11,363 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:11,363 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: picked SnapshotFileV1{lastIncluded=EntryInfo[index=1, term=6], timestamp=2025-08-11T16:03:11.288116700Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-112C4FDC.v1, size=358, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:11,364 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - verifyInstallSnapshotToLaggingFollower starting 04:03:11,474 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 04:03:11,474 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower771: Lagging -1 entries behind leader leader769 and reached 0 (of expected -1) 04:03:11,474 PM [test-pekko.actor.default-dispatcher-7] [WARN] AbstractLeader - leader769 (Leader) : handleAppendEntriesReply delayed beyond election timeout, appendEntriesReply : AppendEntriesReply{term=6, success=true, followerId=follower771, logLastIndex=0, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, timeSinceLastActivity : 3199, lastApplied : 2, commitIndex : 2 04:03:11,474 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractLeader - leader769 (Leader): Initiating install snapshot to follower follower771: follower nextIndex: 1, leader snapshotIndex: 1, leader lastIndex: 2, leader log size: 1 04:03:11,474 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader769: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=2, lastAppliedTerm=6, lastIndex=2, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower771 04:03:11,474 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - leader769: last applied journal index prior to capture: 3 04:03:11,474 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader769: support() called 04:03:11,474 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower771: Lagging -1 entries behind leader leader769 and reached 0 (of expected -1) 04:03:11,475 PM [leader769-%d695321] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:11,475 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:11,475 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Persising snapshot at EntryInfo[index=2, term=6]/EntryInfo[index=2, term=6] 04:03:11,475 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader769: support() called 04:03:11,476 PM [leader769-%d695322] [DEBUG] RaftStorage - leader769: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-1C627D0A.tmp 04:03:11,477 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: Removed in-memory snapshotted entries, adjusted snapshotIndex: 1 and term: 6 04:03:11,477 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): sendInstallSnapshot 04:03:11,477 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] LeaderInstallSnapshotState - leader769 (Leader): Snapshot 310 bytes, total chunks to send: 1 04:03:11,477 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] LeaderInstallSnapshotState - leader769 (Leader): Next chunk: total length=310, offset=0, size=310, hashCode=446856037 04:03:11,477 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): next snapshot chunk size for follower follower771: 310 04:03:11,477 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower771 (Follower): handleInstallSnapshot: InstallSnapshot{term=6, leaderId=leader769, lastIncludedIndex=2, lastIncludedTerm=6, datasize=310, chunk=1/1, lastChunkHashCode=OptionalInt[-1]} 04:03:11,477 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower771: Lagging 2 entries behind leader leader769 and reached 0 (of expected -1) 04:03:11,477 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotTracker - follower771 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 04:03:11,477 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): InstallSnapshot sent to follower Vector(user, follower771), Chunk: 1/1 04:03:11,477 PM [test-pekko.actor.default-dispatcher-7] [INFO] Follower - follower771 (Follower): Snapshot received from leader: leader769 04:03:11,477 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - follower771: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=310}} 04:03:11,477 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower771: support() called 04:03:11,478 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower771: Converted InstallSnapshot from leader: leader769 to state 04:03:11,478 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower771: last applied journal index prior to persisting applied snapshot: 1 04:03:11,478 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower771: support() called 04:03:11,478 PM [follower771-%d466036] [DEBUG] RaftStorage - follower771: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower771/snapshot-00000000689A143F-1C83E2D9.tmp 04:03:11,480 PM [leader769-%d695322] [DEBUG] RaftStorage - leader769: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-1C627D0A.v1 04:03:11,480 PM [leader769-%d695322] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-1C627D0A.v1 04:03:11,480 PM [leader769-%d695322] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-112C4FDC.v1 04:03:11,480 PM [leader769-%d695322] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties 04:03:11,480 PM [leader769-%d695322] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:11,480 PM [leader769-%d695322] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:11,481 PM [leader769-%d695322] [DEBUG] RaftStorage - leader769: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=1, term=6], timestamp=2025-08-11T16:03:11.288116700Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-112C4FDC.v1, size=358, entryCompress=NONE, stateCompress=NONE} 04:03:11,481 PM [leader769-%d695322] [DEBUG] RaftStorage - leader769: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:11.476216586Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-1C627D0A.v1, size=374, entryCompress=NONE, stateCompress=NONE} 04:03:11,481 PM [leader769-%d695322] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:11,481 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:11,481 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - leader769: snapshot is durable as of 2025-08-11T16:03:11.476216586Z 04:03:11,481 PM [follower771-%d466036] [DEBUG] RaftStorage - follower771: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower771/snapshot-00000000689A143F-1C83E2D9.v1 04:03:11,481 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader769: Snapshot success, discarding journal entries up to 3 04:03:11,481 PM [follower771-%d466036] [DEBUG] RaftStorage - follower771: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower771/snapshot-00000000689A143F-1C83E2D9.v1 04:03:11,481 PM [follower771-%d466036] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower771/TermInfo.properties 04:03:11,481 PM [follower771-%d466036] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower771/journal-v1-1.log 04:03:11,481 PM [follower771-%d466036] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower771/metadata-v1-1.log 04:03:11,481 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 189.0 ms 04:03:11,481 PM [follower771-%d466036] [DEBUG] RaftStorage - follower771: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:11.478405337Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/follower771/snapshot-00000000689A143F-1C83E2D9.v1, size=374, entryCompress=NONE, stateCompress=NONE} 04:03:11,481 PM [follower771-%d466036] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:11,481 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:11,481 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - follower771: snapshot is durable as of 2025-08-11T16:03:11.478405337Z 04:03:11,481 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower771: support() called 04:03:11,481 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=4, applyTo=3] 04:03:11,482 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: discarded entries up to (and including) 3 04:03:11,482 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:11,482 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower771: applySnapshot called 04:03:11,482 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 345.1 μs 04:03:11,482 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:11,482 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower771 (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=6, followerId=follower771, chunkIndex=1, success=true} 04:03:11,482 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:11,482 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower771: Snapshot success, discarding journal entries up to 1 04:03:11,482 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorSnapshotMessageSupport - follower771: SnapshotComplete received 04:03:11,482 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 3.327 s 04:03:11,482 PM [follower771-writer-147] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=2, applyTo=1] 04:03:11,482 PM [follower771-writer-147] [DEBUG] EntryJournalV1 - follower771: discarded entries up to (and including) 1 04:03:11,482 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:11,482 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 135.2 μs 04:03:11,482 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:11,482 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:11,482 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorSnapshotMessageSupport - leader769: SnapshotComplete received 04:03:11,482 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader769 (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=6, followerId=follower771, chunkIndex=1, success=true} 04:03:11,482 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractLeader - leader769 (Leader): Snapshot successfully installed on follower follower771 (last chunk 1) - matchIndex set to 2, nextIndex set to 3 04:03:11,594 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 04:03:11,594 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - follower770: fakeSnapshot purging log to 1 for term 6 04:03:11,594 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 04:03:11,714 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 04:03:11,714 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 04:03:11,719 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-1C627D0A.v1 04:03:11,720 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties 04:03:11,720 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:11,720 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:11,720 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: picked SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:11.476216586Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-1C627D0A.v1, size=374, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:11,833 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 04:03:11,833 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 04:03:11,872 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - verifyInstallSnapshotToLaggingFollower complete 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader769: Persist data index=3 term=6 command=three 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=three}, logIndex: 3, isSendImmediate: true 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 3, leaderLastIndex: 3, leaderSnapShotIndex: 1 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 3 is present for follower follower771 04:03:11,873 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 391.1 ms 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower771: AppendEntries{term=6, leaderId=leader769, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 3, leaderLastIndex: 3, leaderSnapShotIndex: 1 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 3 is present for follower follower770 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): Append entry to log three 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:11,873 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=2, prevLogTerm=6, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=6, command=three, pending=true}]} 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): Log size is now 2 04:03:11,873 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Number of entries to be appended = 1 04:03:11,873 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=3, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:11,873 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Append entry to log three 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 04:03:11,873 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower771: Append log entry and persist JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:11,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower770: awaiting more completions to resolve 1 synchronized callback(s) 04:03:11,873 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Log size is now 1 04:03:11,873 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 3, nextIndex: 4 04:03:11,873 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower771, logLastIndex=3, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:11,873 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 2 (of expected -1) 04:03:11,873 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 4, leaderLastIndex: 3, leaderSnapShotIndex: 1 04:03:11,873 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower771 updated: matchIndex: 3, nextIndex: 4 04:03:11,873 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: awaiting more completions to resolve 1 synchronized callback(s) 04:03:11,873 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 4, leaderLastIndex: 3, leaderSnapShotIndex: 1 04:03:11,873 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 518.7 ms 04:03:11,873 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 391.6 ms 04:03:11,876 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:11,876 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 2.111 ms 04:03:11,876 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:11,876 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower771: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=false} 04:03:11,876 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 2.901 ms 04:03:11,876 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=3, term=6, command=three, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@515c1674} 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: synchronized callbacks after 2.385 ms 04:03:11,876 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=true} 04:03:11,876 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 2.316 ms 04:03:11,876 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:11,876 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2 04:03:11,876 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=3, term=6, command=three, pending=false} 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 3 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 2 to 3 04:03:11,876 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower770: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=3, term=6, command=three, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@6111a2ad} 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader769: Applying state for log index 3 data three 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader769: applyState called: three 04:03:11,876 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower770: synchronized callbacks after 2.692 ms 04:03:11,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader769: fakeSnapshot purging log to 2 for term 6 04:03:11,876 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 363.7 μs 04:03:11,876 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=4, applyTo=4] 04:03:11,876 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:11,876 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:11,876 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 224.2 μs 04:03:11,876 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:12,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Commit index set to 3 04:03:12,74 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): Commit index set to 3 04:03:12,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 3 (of expected -1) 04:03:12,74 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 3 (of expected -1) 04:03:12,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): 2, lastIndex(): 3 04:03:12,74 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): 2, lastIndex(): 3 04:03:12,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower771 (Follower): Setting last applied to 3 04:03:12,74 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 3 04:03:12,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower771: Moving last applied index from 2 to 3 04:03:12,74 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 2 to 3 04:03:12,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower771: Applying state for log index 3 data three 04:03:12,74 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower770: Applying state for log index 3 data three 04:03:12,74 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower771: applyState called: three 04:03:12,74 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower770: applyState called: three 04:03:12,75 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower770: fakeSnapshot purging log to 2 for term 6 04:03:12,75 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 199.0 ms 04:03:12,75 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 198.8 ms 04:03:12,75 PM [follower771-writer-147] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=2, applyTo=1] 04:03:12,75 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:12,75 PM [follower770-writer-146] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=4] 04:03:12,75 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 244.0 μs 04:03:12,75 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:12,75 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:12,75 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 305.5 μs 04:03:12,75 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:12,75 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:12,75 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:12,75 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - verifyReplicationsAndSnapshotWithNoLaggingAfterInstallSnapshot starting: replicatedToAllIndex: 2 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader769: Persist data index=4 term=6 command=four 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=four}, logIndex: 4, isSendImmediate: true 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 4, leaderLastIndex: 4, leaderSnapShotIndex: 2 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 4 is present for follower follower771 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower771: AppendEntries{term=6, leaderId=leader769, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=2, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 4, leaderLastIndex: 4, leaderSnapShotIndex: 2 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 4 is present for follower follower770 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=2, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=2, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): Number of entries to be appended = 1 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): After cleanup, lastIndex: 3, entries to be added from: 0 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): Append entry to log four 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower771: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): Log size is now 2 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower771 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower771, logLastIndex=4, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 3 (of expected -1) 04:03:12,76 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower771: awaiting more completions to resolve 1 synchronized callback(s) 04:03:12,76 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 199.9 ms 04:03:12,77 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 1.555 ms 04:03:12,77 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=3, prevLogTerm=6, leaderCommit=3, replicatedToAllIndex=2, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=6, command=four, pending=true}]} 04:03:12,77 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 04:03:12,77 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 3, entries to be added from: 0 04:03:12,77 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): Append entry to log four 04:03:12,77 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:12,77 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): Log size is now 2 04:03:12,77 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=4, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:12,77 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 3 (of expected -1) 04:03:12,77 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower771 updated: matchIndex: 4, nextIndex: 5 04:03:12,77 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 1.859 ms 04:03:12,77 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower770: awaiting more completions to resolve 1 synchronized callback(s) 04:03:12,77 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 5, leaderLastIndex: 4, leaderSnapShotIndex: 2 04:03:12,77 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 4, nextIndex: 5 04:03:12,77 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 5, leaderLastIndex: 4, leaderSnapShotIndex: 2 04:03:12,113 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:12,113 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:12,113 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:12,113 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 36.22 ms 04:03:12,113 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:12,113 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 36.94 ms 04:03:12,113 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 36.63 ms 04:03:12,113 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:12,113 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:12,113 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:12,113 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:12,113 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower771: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=false} 04:03:12,113 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=true} 04:03:12,113 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower771: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=4, term=6, command=four, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@7bcccff6} 04:03:12,114 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower771: synchronized callbacks after 37.26 ms 04:03:12,114 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:12,114 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader769: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=3, lastAppliedTerm=6, lastIndex=4, lastTerm=6, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=1, mandatoryTrim=false] 04:03:12,114 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=4, term=6, command=four, pending=false} 04:03:12,114 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - leader769: captured snapshot at lastSequenceNumber: 4 04:03:12,114 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower770: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=4, term=6, command=four, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@1aaf10eb} 04:03:12,114 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader769: Persising snapshot at EntryInfo[index=3, term=6]/EntryInfo[index=4, term=6] 04:03:12,114 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower770: synchronized callbacks after 36.62 ms 04:03:12,114 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader769: support() called 04:03:12,114 PM [test-pekko.actor.default-dispatcher-7] [INFO] SnapshotManager - leader769: Removed in-memory snapshotted entries, adjusted snapshotIndex: 2 and term: 6 04:03:12,114 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 4, lastAppliedIndex: 3 04:03:12,114 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 4 04:03:12,114 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 3 to 4 04:03:12,114 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader769: Applying state for log index 4 data four 04:03:12,114 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader769: applyState called: four 04:03:12,115 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 1.169 ms 04:03:12,115 PM [leader769-%d695323] [DEBUG] RaftStorage - leader769: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A1440-06D47B4D.tmp 04:03:12,115 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=4, applyTo=5] 04:03:12,115 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:12,115 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 150.1 μs 04:03:12,115 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:12,115 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:12,160 PM [leader769-%d695323] [DEBUG] RaftStorage - leader769: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A1440-06D47B4D.v1 04:03:12,160 PM [leader769-%d695323] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A1440-06D47B4D.v1 04:03:12,160 PM [leader769-%d695323] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-1C627D0A.v1 04:03:12,160 PM [leader769-%d695323] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties 04:03:12,160 PM [leader769-%d695323] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:12,160 PM [leader769-%d695323] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:12,160 PM [leader769-%d695323] [DEBUG] RaftStorage - leader769: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=2, term=6], timestamp=2025-08-11T16:03:11.476216586Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A143F-1C627D0A.v1, size=374, entryCompress=NONE, stateCompress=NONE} 04:03:12,160 PM [leader769-%d695323] [DEBUG] RaftStorage - leader769: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=3, term=6], timestamp=2025-08-11T16:03:12.114588493Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A1440-06D47B4D.v1, size=392, entryCompress=NONE, stateCompress=NONE} 04:03:12,161 PM [leader769-%d695323] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:12,161 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:12,161 PM [test-pekko.actor.default-dispatcher-17] [INFO] SnapshotManager - leader769: snapshot is durable as of 2025-08-11T16:03:12.114588493Z 04:03:12,161 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SnapshotManager - leader769: Snapshot success, discarding journal entries up to 4 04:03:12,161 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorSnapshotMessageSupport - leader769: SnapshotComplete received 04:03:12,161 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 46.02 ms 04:03:12,161 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=5, applyTo=5] 04:03:12,161 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: discarded entries up to (and including) 4 04:03:12,161 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:12,161 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 204.5 μs 04:03:12,161 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:12,161 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:12,193 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): Commit index set to 4 04:03:12,193 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 4 (of expected -1) 04:03:12,193 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower771 (Follower): Commit index set to 4 04:03:12,193 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 4, context.getLastApplied(): 3, lastIndex(): 4 04:03:12,193 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 4 04:03:12,193 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 3 to 4 04:03:12,193 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 4 (of expected -1) 04:03:12,193 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower770: Applying state for log index 4 data four 04:03:12,193 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower770: applyState called: four 04:03:12,193 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower771 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 4, context.getLastApplied(): 3, lastIndex(): 4 04:03:12,193 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower771 (Follower): Setting last applied to 4 04:03:12,193 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower771: Moving last applied index from 3 to 4 04:03:12,193 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower771: Applying state for log index 4 data four 04:03:12,193 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower771: applyState called: four 04:03:12,193 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - leader769: fakeSnapshot purging log to 3 for term 6 04:03:12,193 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 80.00 ms 04:03:12,193 PM [follower770-writer-146] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=5] 04:03:12,193 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:12,193 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 80.02 ms 04:03:12,193 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:12,193 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 134.8 μs 04:03:12,194 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:12,194 PM [follower771-writer-147] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=2, applyTo=2] 04:03:12,194 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:12,194 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:12,194 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 282.8 μs 04:03:12,194 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:12,277 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A1440-06D47B4D.v1 04:03:12,277 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties 04:03:12,277 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:12,277 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:12,277 PM [awaitility-thread] [DEBUG] RaftStorage - leader769: picked SnapshotFileV1{lastIncluded=EntryInfo[index=3, term=6], timestamp=2025-08-11T16:03:12.114588493Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A1440-06D47B4D.v1, size=392, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:12,279 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader769: Persist data index=5 term=6 command=five 04:03:12,279 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=five}, logIndex: 5, isSendImmediate: true 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 5, leaderLastIndex: 5, leaderSnapShotIndex: 3 04:03:12,280 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 118.6 ms 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 5 is present for follower follower771 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower771: AppendEntries{term=6, leaderId=leader769, prevLogIndex=4, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=3, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 5, leaderLastIndex: 5, leaderSnapShotIndex: 3 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 5 is present for follower follower770 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=4, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=3, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader769: Persist data index=6 term=6 command=six 04:03:12,280 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=4, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=3, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader769: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=true} 04:03:12,280 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Number of entries to be appended = 1 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Replicate message: identifier: MockIdentifier{value=six}, logIndex: 6, isSendImmediate: true 04:03:12,280 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): After cleanup, lastIndex: 4, entries to be added from: 0 04:03:12,280 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Append entry to log five 04:03:12,280 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower771: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 5, leaderLastIndex: 6, leaderSnapShotIndex: 3 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 5 is present for follower follower771 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 5, leaderLastIndex: 6, leaderSnapShotIndex: 3 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 5 is present for follower follower770 04:03:12,280 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Log size is now 3 04:03:12,280 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower771, logLastIndex=5, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=4, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=3, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=5, term=6, command=five, pending=true}]} 04:03:12,280 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 4 (of expected -1) 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 04:03:12,280 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower771: fakeSnapshot purging log to 3 for term 6 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 4, entries to be added from: 0 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): Append entry to log five 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:12,280 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: awaiting more completions to resolve 1 synchronized callback(s) 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): Log size is now 3 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=5, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:12,280 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower771 updated: matchIndex: 5, nextIndex: 6 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 4 (of expected -1) 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower770: fakeSnapshot purging log to 3 for term 6 04:03:12,280 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower770: awaiting more completions to resolve 1 synchronized callback(s) 04:03:12,280 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 6, leaderLastIndex: 6, leaderSnapShotIndex: 3 04:03:12,280 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 6 is present for follower follower771 04:03:12,280 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower771: AppendEntries{term=6, leaderId=leader769, prevLogIndex=5, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=3, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 04:03:12,280 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 86.66 ms 04:03:12,281 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 5, nextIndex: 6 04:03:12,281 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 6, leaderLastIndex: 6, leaderSnapShotIndex: 3 04:03:12,281 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): sendAppendEntries: 6 is present for follower follower770 04:03:12,281 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader769 (Leader): Sending AppendEntries to follower follower770: AppendEntries{term=6, leaderId=leader769, prevLogIndex=5, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=3, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 04:03:12,281 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 87.30 ms 04:03:12,298 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:12,298 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:12,298 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:12,298 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 17.79 ms 04:03:12,298 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 17.46 ms 04:03:12,298 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:12,298 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:12,298 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 18.68 ms 04:03:12,298 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 4.070 μs 04:03:12,298 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:12,298 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:12,298 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=false} 04:03:12,298 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower771: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=false} 04:03:12,298 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=5, term=6, command=five, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@628776da} 04:03:12,298 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower770: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=5, term=6, command=five, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@404f14ca} 04:03:12,298 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: synchronized callbacks after 18.26 ms 04:03:12,299 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower770: synchronized callbacks after 18.10 ms 04:03:12,299 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=5, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=3, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 04:03:12,299 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Number of entries to be appended = 1 04:03:12,299 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): handleAppendEntries: AppendEntries{term=6, leaderId=leader769, prevLogIndex=5, prevLogTerm=6, leaderCommit=4, replicatedToAllIndex=3, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=6, term=6, command=six, pending=true}]} 04:03:12,299 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): After cleanup, lastIndex: 5, entries to be added from: 0 04:03:12,299 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Append entry to log six 04:03:12,299 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower771: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=true} 04:03:12,299 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): Number of entries to be appended = 1 04:03:12,299 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): After cleanup, lastIndex: 5, entries to be added from: 0 04:03:12,299 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Log size is now 3 04:03:12,299 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): Append entry to log six 04:03:12,299 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower770: Append log entry and persist JournaledLogEntry{index=6, term=6, command=six, pending=true} 04:03:12,299 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower771, logLastIndex=6, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:12,299 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 4 (of expected -1) 04:03:12,299 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: awaiting more completions to resolve 1 synchronized callback(s) 04:03:12,299 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): Log size is now 3 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:12,299 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): handleAppendEntries returning : AppendEntriesReply{term=6, success=true, followerId=follower770, logLastIndex=6, logLastTerm=6, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=5, term=6, command=five, pending=true} 04:03:12,299 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 4 (of expected -1) 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 5, lastAppliedIndex: 4 04:03:12,299 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower770: awaiting more completions to resolve 1 synchronized callback(s) 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 5 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 4 to 5 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader769: Applying state for log index 5 data five 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader769: applyState called: five 04:03:12,299 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 549.3 μs 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader769: fakeSnapshot purging log to 4 for term 6 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower771 updated: matchIndex: 6, nextIndex: 7 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower771: active: true, followerNextIndex: 7, leaderLastIndex: 6, leaderSnapShotIndex: 4 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader769 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower770 updated: matchIndex: 6, nextIndex: 7 04:03:12,299 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader769 (Leader): Checking sendAppendEntries for follower follower770: active: true, followerNextIndex: 7, leaderLastIndex: 6, leaderSnapShotIndex: 4 04:03:12,299 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 898.0 μs 04:03:12,312 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:12,312 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 12.65 ms 04:03:12,312 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower771: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=false} 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=6, term=6, command=six, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@7a4d9b77} 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower771: synchronized callbacks after 12.95 ms 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): Commit index set to 5 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 5 (of expected -1) 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower771 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 5, context.getLastApplied(): 4, lastIndex(): 6 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower771 (Follower): Setting last applied to 5 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower771: Moving last applied index from 4 to 5 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower771: Applying state for log index 5 data five 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower771: applyState called: five 04:03:12,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower771: fakeSnapshot purging log to 4 for term 6 04:03:12,312 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 399.9 μs 04:03:12,312 PM [follower771-writer-147] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=2, applyTo=3] 04:03:12,312 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:12,312 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 109.5 μs 04:03:12,312 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:12,312 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:12,313 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:12,313 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 13.58 ms 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower770: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=false} 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower770: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=6, term=6, command=six, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08c3380@7f216c02} 04:03:12,313 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower770: synchronized callbacks after 14.09 ms 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): Commit index set to 5 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 5 (of expected -1) 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 5, context.getLastApplied(): 4, lastIndex(): 6 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 5 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 4 to 5 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower770: Applying state for log index 5 data five 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower770: applyState called: five 04:03:12,313 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:12,313 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 14.72 ms 04:03:12,313 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 2.780 μs 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - follower770: fakeSnapshot purging log to 4 for term 6 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader769: persist complete JournaledLogEntry{index=6, term=6, command=six, pending=true} 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader769 (Leader): Applying to log - commitIndex: 6, lastAppliedIndex: 5 04:03:12,313 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=5, applyTo=6] 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader769 (Leader): Setting last applied to 6 04:03:12,313 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 5 to 6 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader769: Applying state for log index 6 data six 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader769: applyState called: six 04:03:12,313 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 173.3 μs 04:03:12,313 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:12,313 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 496.3 μs 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SnapshotManager - leader769: fakeSnapshot purging log to 5 for term 6 04:03:12,313 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:12,313 PM [follower770-writer-146] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=6] 04:03:12,314 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:12,314 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:12,314 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 69.20 μs 04:03:12,314 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:12,314 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 168.2 μs 04:03:12,314 PM [leader769-writer-148] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=5, applyTo=7] 04:03:12,314 PM [leader769-writer-148] [DEBUG] RaftStorageCompleter - leader769: 1 completion(s) pending 04:03:12,314 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 120.5 μs 04:03:12,314 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader769: running 1 completion(s) 04:03:12,314 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:12,433 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower771 (Follower): Commit index set to 6 04:03:12,433 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower771: Lagging 0 entries behind leader leader769 and reached 6 (of expected -1) 04:03:12,433 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower771 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 6, context.getLastApplied(): 5, lastIndex(): 6 04:03:12,433 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): Commit index set to 6 04:03:12,433 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower771 (Follower): Setting last applied to 6 04:03:12,433 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower771: Moving last applied index from 5 to 6 04:03:12,434 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower770: Lagging 0 entries behind leader leader769 and reached 6 (of expected -1) 04:03:12,434 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower771: Applying state for log index 6 data six 04:03:12,434 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower770 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 6, context.getLastApplied(): 5, lastIndex(): 6 04:03:12,434 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower771: applyState called: six 04:03:12,434 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower770 (Follower): Setting last applied to 6 04:03:12,434 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from 5 to 6 04:03:12,434 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower770: Applying state for log index 6 data six 04:03:12,434 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower770: applyState called: six 04:03:12,434 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SnapshotManager - follower771: fakeSnapshot purging log to 5 for term 6 04:03:12,434 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower770: fakeSnapshot purging log to 5 for term 6 04:03:12,434 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 121.6 ms 04:03:12,434 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 120.4 ms 04:03:12,434 PM [follower771-writer-147] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=2, applyTo=4] 04:03:12,434 PM [follower771-writer-147] [DEBUG] RaftStorageCompleter - follower771: 1 completion(s) pending 04:03:12,434 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 208.0 μs 04:03:12,434 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 04:03:12,434 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower771: running 1 completion(s) 04:03:12,434 PM [follower770-writer-146] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=7] 04:03:12,434 PM [follower770-writer-146] [DEBUG] RaftStorageCompleter - follower770: 1 completion(s) pending 04:03:12,434 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 223.0 μs 04:03:12,434 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower770: running 1 completion(s) 04:03:12,434 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: waiting to receive commands 04:03:12,483 PM [main] [INFO] InMemoryJournal - Journal log for leader769: 04:03:12,485 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - verifyReplicationsAndSnapshotWithNoLaggingAfterInstallSnapshot ending 04:03:12,485 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - verifyLeaderRecoveryAfterReinstatement starting: lastIndex: 6, snapshotIndex: 3, firstJournalEntryIndex: 4 04:03:12,496 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EnabledRaftStorage - leader769: terminating thread VirtualThread[#945,leader769-writer-148]/waiting 04:03:12,496 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: received 1 commands after 182.3 ms 04:03:12,496 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 15.93 μs 04:03:12,496 PM [leader769-writer-148] [DEBUG] JournalWriteTask - leader769: journal writer stopped 04:03:12,496 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader769: journal closed 04:03:12,496 PM [test-pekko.actor.default-dispatcher-17] [INFO] EnabledRaftStorage - leader769: journal closed 04:03:12,496 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader769: stopped executor with 0 remaining tasks 04:03:12,496 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader769: stopped executor with 0 remaining tasks [WARN] [08/11/2025 16:03:12.497] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader769]: org.opendaylight.controller.cluster.common.actor.Monitor@23251342 04:03:12,498 PM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - leader769: Starting recovery with journal batch size 1000 04:03:12,498 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader769: started executor 04:03:12,498 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader769: started executor 04:03:12,499 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader769: metadata journal open: replayFrom=5 applyTo=7 04:03:12,499 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader769: journal open: firstIndex=1 lastIndex=7 replayFrom=5 fileEntries=0 04:03:12,499 PM [test-pekko.actor.default-dispatcher-17] [INFO] EnabledRaftStorage - leader769: journal open: applyTo=7 04:03:12,499 PM [leader769-writer-149] [DEBUG] JournalWriteTask - leader769: journal writer started 04:03:12,499 PM [leader769-writer-149] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 04:03:12,500 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=6, votedFor=leader769 04:03:12,501 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A1440-06D47B4D.v1 04:03:12,501 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/TermInfo.properties 04:03:12,501 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/journal-v1-1.log 04:03:12,501 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/metadata-v1-1.log 04:03:12,501 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader769: picked SnapshotFileV1{lastIncluded=EntryInfo[index=3, term=6], timestamp=2025-08-11T16:03:12.114588493Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest15084952947945874642/leader769/snapshot-00000000689A1440-06D47B4D.v1, size=392, entryCompress=NONE, stateCompress=NONE} as the latest file 04:03:12,501 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader769: support() called 04:03:12,502 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - leader769: initializing from snapshot taken at 2025-08-11T16:03:12.114588493Z 04:03:12,502 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=6, votedFor=leader769 04:03:12,502 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for leader769 in 88.72 μs: snapshotIndex=3, snapshotTerm=6, journal-size=0 04:03:12,502 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader769: Recovery completed in 354.1 μs - Switching actor to Follower - last log index = 3, last log term = 6, snapshot index = 3, snapshot term = 6, journal size = 0 04:03:12,502 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - leader769: restoring local TermInfo{term=6, votedFor=leader769} 04:03:12,502 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=6, votedFor=leader769 04:03:12,503 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader769: Pekko recovery completed and cannot restore from snapshot 04:03:12,503 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader769: starting journal recovery 04:03:12,503 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from -1 to 3 04:03:12,504 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader769: applying entries up to 7 04:03:12,504 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader769: recovered journal DefaultLogEntry[index=4, term=6, command=four] 04:03:12,504 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader769: recovered journal DefaultLogEntry[index=5, term=6, command=five] 04:03:12,504 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader769: recovered journal DefaultLogEntry[index=6, term=6, command=six] 04:03:12,504 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from 3 to 6 04:03:12,504 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader769: journal recovery completedin 656.5 μs with journalIndex=5 04:03:12,504 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader769: Journal recovery completed and cannot restore from snapshot 04:03:12,504 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - verifyLeaderRecoveryAfterReinstatement ending 04:03:12,504 PM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotTriggeredByMemoryThresholdExceeded ending 04:03:12,504 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower770-collector772#98965384] 04:03:12,504 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader769 (Follower): Received TimeoutNow - switching to Candidate 04:03:12,504 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader769 (Candidate): Election: Candidate has following voting peers: [] 04:03:12,507 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=7, votedFor=leader769 04:03:12,507 PM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - leader769 (Candidate): Starting new election term 7 04:03:12,507 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader769 (Follower) :- Switching from behavior Follower to Candidate, election term: 7 04:03:12,507 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader769 (Candidate): Received ElectionTimeout 04:03:12,507 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader769 (Leader): Election: Leader has following peers: [] 04:03:12,507 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader769 (Candidate) :- Switching from behavior Candidate to Leader, election term: 7 04:03:12,515 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower770] 04:03:12,515 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - follower770: terminating thread VirtualThread[#943,follower770-writer-146]/waiting 04:03:12,515 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: received 1 commands after 80.87 ms 04:03:12,515 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 15.35 μs 04:03:12,515 PM [follower770-writer-146] [DEBUG] JournalWriteTask - follower770: journal writer stopped 04:03:12,515 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower770: journal closed 04:03:12,515 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - follower770: journal closed 04:03:12,515 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower770: stopped executor with 0 remaining tasks 04:03:12,515 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower770: stopped executor with 0 remaining tasks 04:03:12,516 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower771-collector773#-1493363821] 04:03:12,516 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower771] 04:03:12,516 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - follower771: terminating thread VirtualThread[#944,follower771-writer-147]/waiting 04:03:12,516 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: received 1 commands after 81.73 ms 04:03:12,516 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 9.321 μs 04:03:12,516 PM [follower771-writer-147] [DEBUG] JournalWriteTask - follower771: journal writer stopped 04:03:12,516 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower771: journal closed 04:03:12,516 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - follower771: journal closed 04:03:12,516 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower771: stopped executor with 0 remaining tasks 04:03:12,516 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower771: stopped executor with 0 remaining tasks 04:03:12,516 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader769-collector774#937885957] 04:03:12,516 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader769] 04:03:12,516 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader769-collector775#1098286756] 04:03:12,516 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader769] 04:03:12,516 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - leader769: terminating thread VirtualThread[#958,leader769-writer-149]/waiting 04:03:12,517 PM [leader769-writer-149] [DEBUG] JournalWriteTask - leader769: received 1 commands after 17.41 ms 04:03:12,517 PM [leader769-writer-149] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 9.321 μs 04:03:12,517 PM [leader769-writer-149] [DEBUG] JournalWriteTask - leader769: journal writer stopped 04:03:12,517 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader769: journal closed 04:03:12,517 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader769: journal closed 04:03:12,517 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader769: stopped executor with 0 remaining tasks 04:03:12,517 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader769: stopped executor with 0 remaining tasks [INFO] [08/11/2025 16:03:12.518] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [08/11/2025 16:03:12.519] [test-pekko.actor.default-dispatcher-17] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.170.10:17355) 04:03:12,519 PM [test-pekko.actor.default-dispatcher-17] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [INFO] [08/11/2025 16:03:12.519] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.10:17355] - Exiting completed [INFO] [08/11/2025 16:03:12.519] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.10:17355] - Shutting down... 04:03:12,519 PM [test-pekko.actor.default-dispatcher-17] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.10:17355] - Exiting completed 04:03:12,519 PM [test-pekko.actor.default-dispatcher-17] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.10:17355] - Shutting down... [INFO] [08/11/2025 16:03:12.519] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.10:17355] - Successfully shut down 04:03:12,519 PM [test-pekko.actor.default-dispatcher-17] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.10:17355] - Successfully shut down 04:03:12,521 PM [test-pekko.actor.default-dispatcher-7] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. [INFO] [08/11/2025 16:03:12.521] [test-pekko.remote.default-remote-dispatcher-10] [pekko://test@10.30.170.10:17355/system/remoting-terminator] Shutting down remote daemon. [INFO] [08/11/2025 16:03:12.521] [test-pekko.remote.default-remote-dispatcher-10] [pekko://test@10.30.170.10:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 04:03:12,522 PM [test-pekko.actor.default-dispatcher-7] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [08/11/2025 16:03:12.524] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.170.10:17355/system/remoting-terminator] Remoting shut down. 04:03:12,524 PM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [DEBUG] [08/11/2025 16:03:12.524] [test-pekko.actor.internal-dispatcher-2] [EventStream] shutting down: StandardOutLogger