[DEBUG] [03/10/2026 10:36:39.207] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 10:36:39,212 AM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [03/10/2026 10:36:39.213] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [03/10/2026 10:36:39.213] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [03/10/2026 10:36:39.248] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.75:17355] with UID [-3697815842238804604] 10:36:39,248 AM [test-pekko.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.75:17355] with UID [-3697815842238804604] [INFO] [03/10/2026 10:36:39.249] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.75:17355] - Starting up, Pekko version [1.2.1] ... 10:36:39,249 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.75:17355] - Starting up, Pekko version [1.2.1] ... [INFO] [03/10/2026 10:36:39.254] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.75:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 10:36:39,254 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.75:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [03/10/2026 10:36:39.254] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.75:17355] - Started up successfully 10:36:39,255 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.75:17355] - Started up successfully [INFO] [03/10/2026 10:36:39.260] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.75:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [03/10/2026 10:36:39.260] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.75:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 10:36:39,262 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.75:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 10:36:39,262 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.75:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 10:36:39,277 AM [main] [INFO] NonVotingFollowerIntegrationTest - testFollowerLeaderStateChanges 10:36:39,281 AM [main] [DEBUG] PropertiesTermInfoStore - leader16: Set currentTerm=1, votedFor=leader16 10:36:39,284 AM [main] [DEBUG] PropertiesTermInfoStore - follower17: Set currentTerm=1, votedFor=leader16 10:36:39,284 AM [main] [DEBUG] RaftStorage - leader16: starting snapshot writeout to /tmp/junit-5254439000667939916/leader16/snapshot-0000000069AFF437-10F28CD3.tmp 10:36:39,285 AM [main] [DEBUG] RaftStorage - leader16: finished snapshot writeout to /tmp/junit-5254439000667939916/leader16/snapshot-0000000069AFF437-10F28CD3.v1 10:36:39,286 AM [main] [DEBUG] RaftStorage - follower17: starting snapshot writeout to /tmp/junit-5254439000667939916/follower17/snapshot-0000000069AFF437-110BEC49.tmp 10:36:39,287 AM [main] [DEBUG] RaftStorage - follower17: finished snapshot writeout to /tmp/junit-5254439000667939916/follower17/snapshot-0000000069AFF437-110BEC49.v1 10:36:39,289 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/follower17] 10:36:39,291 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower17: Starting recovery with journal batch size 1000 10:36:39,291 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower17: started executor 10:36:39,291 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower17: started executor [WARN] [03/10/2026 10:36:39.292] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower17]: org.opendaylight.controller.cluster.common.actor.Monitor@46e285a0 10:36:39,292 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower17: metadata journal open: replayFrom=1 applyTo=0 10:36:39,293 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower17: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:39,293 AM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - follower17: journal open: applyTo=0 10:36:39,293 AM [follower17-writer-9] [DEBUG] JournalWriteTask - follower17: journal writer started 10:36:39,293 AM [follower17-writer-9] [DEBUG] JournalWriteTask - follower17: waiting to receive commands 10:36:39,293 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower17: recovering to non-persistent [INFO] [pekkoDeadLetter][03/10/2026 10:36:39.292] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower17] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:39,293 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower17: Set currentTerm=1, votedFor=leader16 10:36:39,294 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower17: skipping unrecognized file /tmp/junit-5254439000667939916/follower17/TermInfo.properties 10:36:39,294 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower17: skipping unrecognized file /tmp/junit-5254439000667939916/follower17/journal-v1-1.log 10:36:39,294 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower17: selected SNAPSHOT_V1 to handle file /tmp/junit-5254439000667939916/follower17/snapshot-0000000069AFF437-110BEC49.v1 10:36:39,294 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower17: skipping unrecognized file /tmp/junit-5254439000667939916/follower17/metadata-v1-1.log 10:36:39,294 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower17: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:39.285994057Z, file=/tmp/junit-5254439000667939916/follower17/snapshot-0000000069AFF437-110BEC49.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:39,294 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Recovery - follower17: initializing from snapshot taken at 2026-03-10T10:36:39.285994057Z 10:36:39,294 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - follower17: Updated server config: isVoting: false, peers: [PeerInfo [id=leader16, address=pekko://test/user/leader16, votingState=VOTING]] 10:36:39,294 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower17: support() called 10:36:39,296 AM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower17] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:39,296 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower17: Moving last applied index from -1 to -1 10:36:39,297 AM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - follower17: Recovery completed in in 3.041 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:39,297 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower17: Recovery completed and cannot restore from snapshot 10:36:39,399 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/leader16] [WARN] [03/10/2026 10:36:39.400] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader16]: org.opendaylight.controller.cluster.common.actor.Monitor@dbe3524 [INFO] [pekkoDeadLetter][03/10/2026 10:36:39.400] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader16] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:39,401 AM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader16] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:39,402 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - leader16: Starting recovery with journal batch size 1000 10:36:39,402 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader16: started executor 10:36:39,402 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader16: started executor 10:36:39,403 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader16: metadata journal open: replayFrom=1 applyTo=0 10:36:39,404 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader16: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:39,404 AM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - leader16: journal open: applyTo=0 10:36:39,404 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader16: recovering to non-persistent 10:36:39,404 AM [leader16-writer-10] [DEBUG] JournalWriteTask - leader16: journal writer started 10:36:39,404 AM [leader16-writer-10] [DEBUG] JournalWriteTask - leader16: waiting to receive commands 10:36:39,405 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader16: Set currentTerm=1, votedFor=leader16 10:36:39,405 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader16: selected SNAPSHOT_V1 to handle file /tmp/junit-5254439000667939916/leader16/snapshot-0000000069AFF437-10F28CD3.v1 10:36:39,405 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader16: skipping unrecognized file /tmp/junit-5254439000667939916/leader16/TermInfo.properties 10:36:39,405 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader16: skipping unrecognized file /tmp/junit-5254439000667939916/leader16/journal-v1-1.log 10:36:39,405 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader16: skipping unrecognized file /tmp/junit-5254439000667939916/leader16/metadata-v1-1.log 10:36:39,405 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader16: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:39.284331219Z, file=/tmp/junit-5254439000667939916/leader16/snapshot-0000000069AFF437-10F28CD3.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:39,405 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Recovery - leader16: initializing from snapshot taken at 2026-03-10T10:36:39.284331219Z 10:36:39,406 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - leader16: Updated server config: isVoting: true, peers: [PeerInfo [id=follower17, address=pekko://test/user/follower17, votingState=NON_VOTING]] 10:36:39,406 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader16: support() called 10:36:39,406 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader16: Moving last applied index from -1 to -1 10:36:39,406 AM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - leader16: Recovery completed in in 1.764 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:39,406 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader16: Recovery completed and cannot restore from snapshot 10:36:39,835 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - leader16 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.75:17355, Removed) 10:36:39,835 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - leader16 (Follower): Received ElectionTimeout - switching to Candidate 10:36:39,835 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader16 (Candidate): Election: Candidate has following voting peers: [] 10:36:39,838 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader16: Set currentTerm=2, votedFor=leader16 10:36:39,838 AM [test-pekko.actor.default-dispatcher-11] [INFO] Candidate - leader16 (Candidate): Starting new election term 2 10:36:39,838 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader16 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 10:36:39,839 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader16 (Candidate): Received ElectionTimeout 10:36:39,839 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader16 (Leader): Election: Leader has following peers: [follower17] 10:36:39,839 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader16 (Leader): Checking sendAppendEntries for follower follower17: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:39,839 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader16 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 10:36:39,840 AM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower17 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader16, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 10:36:39,842 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower17: Set currentTerm=2, votedFor=null 10:36:39,843 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower17: Last sync leader does not match current leader leader16, need to catch up to -1 10:36:39,844 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader16 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower17 updated: matchIndex: -1, nextIndex: 0 10:36:39,844 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader16 (Leader): Checking sendAppendEntries for follower follower17: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/10/2026 10:36:39.845] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower17] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower17, initialSyncDone=false] [INFO] [pekkoDeadLetter][03/10/2026 10:36:39.845] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower17] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower17] was unhandled. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:39,845 AM [test-pekko.actor.default-dispatcher-12] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower17] was unhandled. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:40,196 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower17: Lagging 0 entries behind leader leader16 and reached -1 (of expected -1) 10:36:40,197 AM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower17] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [03/10/2026 10:36:40.198] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower17] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower17, initialSyncDone=true] [INFO] [pekkoDeadLetter][03/10/2026 10:36:40.197] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower17] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower17] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:40,221 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/roleChangeNotifier19#-874669505] 10:36:40,234 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower17-collector20#-778411134] 10:36:40,235 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower17] 10:36:40,235 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EnabledRaftStorage - follower17: terminating thread VirtualThread[#122,follower17-writer-9]/waiting 10:36:40,235 AM [follower17-writer-9] [DEBUG] JournalWriteTask - follower17: received 1 commands after 942.1 ms 10:36:40,236 AM [follower17-writer-9] [DEBUG] JournalWriteTask - follower17: completed 1 commands in 35.96 μs 10:36:40,236 AM [follower17-writer-9] [DEBUG] JournalWriteTask - follower17: journal writer stopped 10:36:40,237 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower17: journal closed 10:36:40,237 AM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - follower17: journal closed 10:36:40,237 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower17: stopped executor with 0 remaining tasks 10:36:40,237 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower17: stopped executor with 0 remaining tasks 10:36:40,237 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader16-collector21#1090704179] 10:36:40,238 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader16] 10:36:40,238 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EnabledRaftStorage - leader16: terminating thread VirtualThread[#123,leader16-writer-10]/waiting 10:36:40,238 AM [leader16-writer-10] [DEBUG] JournalWriteTask - leader16: received 1 commands after 833.6 ms 10:36:40,238 AM [leader16-writer-10] [DEBUG] JournalWriteTask - leader16: completed 1 commands in 22.34 μs 10:36:40,238 AM [leader16-writer-10] [DEBUG] JournalWriteTask - leader16: journal writer stopped 10:36:40,239 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader16: journal closed 10:36:40,239 AM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - leader16: journal closed 10:36:40,239 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader16: stopped executor with 0 remaining tasks 10:36:40,239 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader16: stopped executor with 0 remaining tasks 10:36:40,243 AM [main] [INFO] NonVotingFollowerIntegrationTest - testFollowerResyncWithLessLeaderLogEntriesAfterNonPersistentLeaderRestart starting 10:36:40,245 AM [main] [DEBUG] PropertiesTermInfoStore - leader22: Set currentTerm=1, votedFor=leader22 10:36:40,247 AM [main] [DEBUG] PropertiesTermInfoStore - follower23: Set currentTerm=1, votedFor=leader22 10:36:40,247 AM [main] [DEBUG] RaftStorage - leader22: starting snapshot writeout to /tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF438-0EBA313F.tmp 10:36:40,248 AM [main] [DEBUG] RaftStorage - leader22: finished snapshot writeout to /tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF438-0EBA313F.v1 10:36:40,248 AM [main] [DEBUG] RaftStorage - follower23: starting snapshot writeout to /tmp/junit-17223293662477380498/follower23/snapshot-0000000069AFF438-0ECE43FF.tmp 10:36:40,249 AM [main] [DEBUG] RaftStorage - follower23: finished snapshot writeout to /tmp/junit-17223293662477380498/follower23/snapshot-0000000069AFF438-0ECE43FF.v1 10:36:40,252 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/follower23] [WARN] [03/10/2026 10:36:40.252] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower23]: org.opendaylight.controller.cluster.common.actor.Monitor@50eb6a32 [INFO] [pekkoDeadLetter][03/10/2026 10:36:40.252] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower23] to Actor[pekko://test/user/termination-monitor] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:40,253 AM [test-pekko.actor.default-dispatcher-13] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower23] to Actor[pekko://test/user/termination-monitor] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:40,254 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - follower23: Starting recovery with journal batch size 1000 10:36:40,254 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower23: started executor 10:36:40,254 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower23: started executor 10:36:40,256 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower23: metadata journal open: replayFrom=1 applyTo=0 10:36:40,257 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower23: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:40,257 AM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - follower23: journal open: applyTo=0 10:36:40,257 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower23: recovering to non-persistent 10:36:40,257 AM [follower23-writer-11] [DEBUG] JournalWriteTask - follower23: journal writer started 10:36:40,257 AM [follower23-writer-11] [DEBUG] JournalWriteTask - follower23: waiting to receive commands 10:36:40,257 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower23: Set currentTerm=1, votedFor=leader22 10:36:40,257 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower23: skipping unrecognized file /tmp/junit-17223293662477380498/follower23/TermInfo.properties 10:36:40,257 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower23: skipping unrecognized file /tmp/junit-17223293662477380498/follower23/journal-v1-1.log 10:36:40,257 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower23: skipping unrecognized file /tmp/junit-17223293662477380498/follower23/metadata-v1-1.log 10:36:40,257 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower23: selected SNAPSHOT_V1 to handle file /tmp/junit-17223293662477380498/follower23/snapshot-0000000069AFF438-0ECE43FF.v1 10:36:40,258 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower23: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:40.248398847Z, file=/tmp/junit-17223293662477380498/follower23/snapshot-0000000069AFF438-0ECE43FF.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:40,258 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Recovery - follower23: initializing from snapshot taken at 2026-03-10T10:36:40.248398847Z 10:36:40,258 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - follower23: Updated server config: isVoting: false, peers: [PeerInfo [id=leader22, address=pekko://test/user/leader22, votingState=VOTING]] 10:36:40,258 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower23: support() called 10:36:40,259 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower23: Moving last applied index from -1 to -1 10:36:40,259 AM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - follower23: Recovery completed in in 1.848 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:40,259 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower23: Recovery completed and cannot restore from snapshot 10:36:40,262 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/leader22] [WARN] [03/10/2026 10:36:40.262] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader22]: org.opendaylight.controller.cluster.common.actor.Monitor@981a04b 10:36:40,263 AM [test-pekko.actor.default-dispatcher-13] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader22] to Actor[pekko://test/user/termination-monitor] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][03/10/2026 10:36:40.262] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader22] to Actor[pekko://test/user/termination-monitor] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:40,263 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - leader22: Starting recovery with journal batch size 1000 10:36:40,263 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader22: started executor 10:36:40,263 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader22: started executor 10:36:40,264 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader22: metadata journal open: replayFrom=1 applyTo=0 10:36:40,264 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader22: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:40,264 AM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - leader22: journal open: applyTo=0 10:36:40,264 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader22: recovering to non-persistent 10:36:40,264 AM [leader22-writer-12] [DEBUG] JournalWriteTask - leader22: journal writer started 10:36:40,264 AM [leader22-writer-12] [DEBUG] JournalWriteTask - leader22: waiting to receive commands 10:36:40,264 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader22: Set currentTerm=1, votedFor=leader22 10:36:40,264 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader22: selected SNAPSHOT_V1 to handle file /tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF438-0EBA313F.v1 10:36:40,265 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader22: skipping unrecognized file /tmp/junit-17223293662477380498/leader22/TermInfo.properties 10:36:40,265 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader22: skipping unrecognized file /tmp/junit-17223293662477380498/leader22/journal-v1-1.log 10:36:40,265 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader22: skipping unrecognized file /tmp/junit-17223293662477380498/leader22/metadata-v1-1.log 10:36:40,265 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader22: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:40.247083327Z, file=/tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF438-0EBA313F.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:40,265 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Recovery - leader22: initializing from snapshot taken at 2026-03-10T10:36:40.247083327Z 10:36:40,265 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - leader22: Updated server config: isVoting: true, peers: [PeerInfo [id=follower23, address=pekko://test/user/follower23, votingState=NON_VOTING]] 10:36:40,265 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader22: support() called 10:36:40,265 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader22: Moving last applied index from -1 to -1 10:36:40,266 AM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - leader22: Recovery completed in in 1.178 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:40,266 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader22: Recovery completed and cannot restore from snapshot 10:36:40,715 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader22 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.75:17355, Removed) 10:36:40,715 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader22 (Follower): Received ElectionTimeout - switching to Candidate 10:36:40,715 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader22 (Candidate): Election: Candidate has following voting peers: [] 10:36:40,720 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader22: Set currentTerm=2, votedFor=leader22 10:36:40,720 AM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - leader22 (Candidate): Starting new election term 2 10:36:40,720 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader22 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 10:36:40,721 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader22 (Candidate): Received ElectionTimeout 10:36:40,721 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader22 (Leader): Election: Leader has following peers: [follower23] 10:36:40,721 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:40,722 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader22 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 10:36:40,722 AM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower23 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader22, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 10:36:40,725 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower23: Set currentTerm=2, votedFor=null 10:36:40,725 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader22 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower23 updated: matchIndex: -1, nextIndex: 0 10:36:40,725 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:40,725 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower23: Last sync leader does not match current leader leader22, need to catch up to -1 [WARN] [03/10/2026 10:36:40.726] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower23] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower23, initialSyncDone=false] [INFO] [pekkoDeadLetter][03/10/2026 10:36:40.726] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower23] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower23] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:40,726 AM [test-pekko.actor.default-dispatcher-14] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower23] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:40,777 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader22: Persist data index=0 term=2 command=zero 10:36:40,777 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader22: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 10:36:40,777 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader22: 1 completion(s) pending 10:36:40,777 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 10:36:40,777 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader22 (Leader): Setting last applied to 0 10:36:40,777 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader22: Moving last applied index from -1 to 0 10:36:40,777 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader22: Applying state for log index 0 data zero 10:36:40,777 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader22: applyState called: zero 10:36:40,779 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:40,779 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): sendAppendEntries: 0 is present for follower follower23 10:36:40,779 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Sending AppendEntries to follower follower23: AppendEntries{term=2, leaderId=leader22, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 10:36:40,779 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader22: running 1 completion(s) 10:36:40,779 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader22: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} 10:36:40,780 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader22, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 10:36:40,780 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Number of entries to be appended = 1 10:36:40,780 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader22: Persist data index=1 term=2 command=one 10:36:40,780 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 10:36:40,780 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader22: Append log entry and persist JournaledLogEntry{index=1, term=2, command=one, pending=true} 10:36:40,780 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader22: 1 completion(s) pending 10:36:40,780 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 10:36:40,780 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader22 (Leader): Setting last applied to 1 10:36:40,780 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader22: Moving last applied index from 0 to 1 10:36:40,780 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader22: Applying state for log index 1 data one 10:36:40,780 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader22: applyState called: one 10:36:40,781 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Append entry to log zero 10:36:40,781 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower23: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=false} 10:36:40,781 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 10:36:40,781 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): sendAppendEntries: 0 is present for follower follower23 10:36:40,781 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Sending AppendEntries to follower follower23: AppendEntries{term=2, leaderId=leader22, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}, JournaledLogEntry{index=1, term=2, command=one, pending=true}]} 10:36:40,781 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: 1 completion(s) pending 10:36:40,781 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Log size is now 1 10:36:40,781 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader22: running 1 completion(s) 10:36:40,781 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Commit index set to 0 10:36:40,781 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader22: persist complete JournaledLogEntry{index=1, term=2, command=one, pending=true} 10:36:40,781 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower23, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:40,781 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader22: Persist data index=2 term=2 command=two 10:36:40,781 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader22: Append log entry and persist JournaledLogEntry{index=2, term=2, command=two, pending=true} 10:36:40,781 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower23: Lagging 0 entries behind leader leader22 and reached 0 (of expected -1) 10:36:40,781 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader22: 1 completion(s) pending 10:36:40,782 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 10:36:40,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Replicate message: identifier: MockIdentifier{value=two}, logIndex: 2, isSendImmediate: true 10:36:40,782 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower23 (Follower): Setting last applied to 0 10:36:40,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader22 (Leader): Setting last applied to 2 10:36:40,782 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower23: Moving last applied index from -1 to 0 10:36:40,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader22: Moving last applied index from 1 to 2 10:36:40,782 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower23: Applying state for log index 0 data zero 10:36:40,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader22: Applying state for log index 2 data two 10:36:40,782 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower23: applyState called: zero 10:36:40,782 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader22: applyState called: two 10:36:40,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: true, followerNextIndex: 0, leaderLastIndex: 2, leaderSnapShotIndex: -1 10:36:40,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): sendAppendEntries: 0 is present for follower follower23 10:36:40,782 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: running 1 completion(s) 10:36:40,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Sending AppendEntries to follower follower23: AppendEntries{term=2, leaderId=leader22, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=2, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}, JournaledLogEntry{index=1, term=2, command=one, pending=false}, JournaledLogEntry{index=2, term=2, command=two, pending=true}]} 10:36:40,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader22: running 1 completion(s) 10:36:40,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader22: persist complete JournaledLogEntry{index=2, term=2, command=two, pending=true} 10:36:40,783 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower23 updated: matchIndex: 0, nextIndex: 1 10:36:40,783 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - leader22: fakeSnapshot purging log to 0 for term 2 10:36:40,783 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower23: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 10:36:40,783 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: true, followerNextIndex: 1, leaderLastIndex: 2, leaderSnapShotIndex: 0 10:36:40,783 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=2, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@33583314} 10:36:40,783 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): sendAppendEntries: 1 is present for follower follower23 10:36:40,783 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Sending AppendEntries to follower follower23: AppendEntries{term=2, leaderId=leader22, prevLogIndex=0, prevLogTerm=2, leaderCommit=2, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=one, pending=false}, JournaledLogEntry{index=2, term=2, command=two, pending=false}]} 10:36:40,783 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: synchronized callbacks after 843.3 μs 10:36:40,783 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader22, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}, JournaledLogEntry{index=1, term=2, command=one, pending=false}]} 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Number of entries to be appended = 2 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): matchEntry JournaledLogEntry{index=0, term=2, command=zero, pending=false} is present: existingEntryTerm: 2 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): After cleanup, lastIndex: 0, entries to be added from: 1 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Append entry to log one 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower23: Append log entry and persist JournaledLogEntry{index=1, term=2, command=one, pending=false} 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: 1 completion(s) pending 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Log size is now 2 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Commit index set to 1 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower23, logLastIndex=1, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower23: Lagging 0 entries behind leader leader22 and reached 1 (of expected -1) 10:36:40,784 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 10:36:40,785 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower23 (Follower): Setting last applied to 1 10:36:40,785 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower23 updated: matchIndex: 1, nextIndex: 2 10:36:40,785 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower23: Moving last applied index from 0 to 1 10:36:40,785 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - leader22: fakeSnapshot purging log to 1 for term 2 10:36:40,785 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower23: Applying state for log index 1 data one 10:36:40,785 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower23: applyState called: one 10:36:40,785 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 1 10:36:40,785 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): sendAppendEntries: 2 is present for follower follower23 10:36:40,785 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader22 (Leader): Sending AppendEntries to follower follower23: AppendEntries{term=2, leaderId=leader22, prevLogIndex=1, prevLogTerm=2, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=2, command=two, pending=false}]} 10:36:40,785 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: running 1 completion(s) 10:36:40,785 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower23: persist complete JournaledLogEntry{index=1, term=2, command=one, pending=false} 10:36:40,785 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=2, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@18bc9954} 10:36:40,785 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: synchronized callbacks after 209.6 μs 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader22, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=2, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}, JournaledLogEntry{index=1, term=2, command=one, pending=false}, JournaledLogEntry{index=2, term=2, command=two, pending=false}]} 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Number of entries to be appended = 3 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): matchEntry JournaledLogEntry{index=0, term=2, command=zero, pending=false} is present: existingEntryTerm: 2 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): matchEntry JournaledLogEntry{index=1, term=2, command=one, pending=false} is present: existingEntryTerm: 2 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): After cleanup, lastIndex: 1, entries to be added from: 2 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Append entry to log two 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower23: Append log entry and persist JournaledLogEntry{index=2, term=2, command=two, pending=false} 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: 1 completion(s) pending 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Log size is now 3 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): Commit index set to 2 10:36:40,786 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower23, logLastIndex=2, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:40,786 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader22 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower23 updated: matchIndex: 2, nextIndex: 3 10:36:40,786 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 1 10:36:40,787 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower23: Lagging 0 entries behind leader leader22 and reached 2 (of expected -1) 10:36:40,787 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower23 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 2 10:36:40,787 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower23 (Follower): Setting last applied to 2 10:36:40,787 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower23: Moving last applied index from 1 to 2 10:36:40,787 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower23: Applying state for log index 2 data two 10:36:40,787 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower23: applyState called: two 10:36:40,787 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: running 1 completion(s) 10:36:40,787 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower23: persist complete JournaledLogEntry{index=2, term=2, command=two, pending=false} 10:36:40,787 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=2, command=two, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@36a061b} 10:36:40,787 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower23: synchronized callbacks after 144.3 μs 10:36:40,787 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader22, prevLogIndex=0, prevLogTerm=2, leaderCommit=2, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=one, pending=false}, JournaledLogEntry{index=2, term=2, command=two, pending=false}]} 10:36:40,787 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): Number of entries to be appended = 2 10:36:40,787 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): matchEntry JournaledLogEntry{index=1, term=2, command=one, pending=false} is present: existingEntryTerm: 2 10:36:40,787 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): matchEntry JournaledLogEntry{index=2, term=2, command=two, pending=false} is present: existingEntryTerm: 2 10:36:40,787 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): After cleanup, lastIndex: 2, entries to be added from: 2 10:36:40,787 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): Log size is now 3 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower23, logLastIndex=2, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower23: Lagging 0 entries behind leader leader22 and reached 2 (of expected -1) 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - follower23: fakeSnapshot purging log to 0 for term 2 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader22, prevLogIndex=1, prevLogTerm=2, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=2, command=two, pending=false}]} 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): Number of entries to be appended = 1 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): matchEntry JournaledLogEntry{index=2, term=2, command=two, pending=false} is present: existingEntryTerm: 2 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): After cleanup, lastIndex: 2, entries to be added from: 1 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): Log size is now 2 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower23, logLastIndex=2, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower23: Lagging 0 entries behind leader leader22 and reached 2 (of expected -1) 10:36:40,788 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - follower23: fakeSnapshot purging log to 1 for term 2 [WARN] [03/10/2026 10:36:40.789] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower23] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower23, initialSyncDone=true] 10:36:40,789 AM [test-pekko.actor.default-dispatcher-13] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower23] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][03/10/2026 10:36:40.789] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower23] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower23] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:40,842 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - leader22: terminating thread VirtualThread[#127,leader22-writer-12]/waiting 10:36:40,843 AM [leader22-writer-12] [DEBUG] JournalWriteTask - leader22: received 1 commands after 578.2 ms 10:36:40,843 AM [leader22-writer-12] [DEBUG] JournalWriteTask - leader22: completed 1 commands in 31.48 μs 10:36:40,843 AM [leader22-writer-12] [DEBUG] JournalWriteTask - leader22: journal writer stopped 10:36:40,843 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader22: journal closed 10:36:40,843 AM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader22: journal closed 10:36:40,843 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: stopped executor with 0 remaining tasks 10:36:40,844 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: stopped executor with 0 remaining tasks 10:36:40,848 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/leader22] [WARN] [03/10/2026 10:36:40.848] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader22]: org.opendaylight.controller.cluster.common.actor.Monitor@6b61c5fa [INFO] [pekkoDeadLetter][03/10/2026 10:36:40.848] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader22] to Actor[pekko://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:40,849 AM [test-pekko.actor.default-dispatcher-13] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader22] to Actor[pekko://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:40,849 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader22: Starting recovery with journal batch size 1000 10:36:40,849 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: started executor 10:36:40,849 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: started executor 10:36:40,850 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader22: metadata journal open: replayFrom=1 applyTo=0 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader22: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader22: journal open: applyTo=0 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader22: recovering to non-persistent 10:36:40,851 AM [leader22-writer-13] [DEBUG] JournalWriteTask - leader22: journal writer started 10:36:40,851 AM [leader22-writer-13] [DEBUG] JournalWriteTask - leader22: waiting to receive commands 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader22: Set currentTerm=2, votedFor=leader22 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: selected SNAPSHOT_V1 to handle file /tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF438-0EBA313F.v1 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: skipping unrecognized file /tmp/junit-17223293662477380498/leader22/TermInfo.properties 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: skipping unrecognized file /tmp/junit-17223293662477380498/leader22/journal-v1-1.log 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: skipping unrecognized file /tmp/junit-17223293662477380498/leader22/metadata-v1-1.log 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:40.247083327Z, file=/tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF438-0EBA313F.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Recovery - leader22: initializing from snapshot taken at 2026-03-10T10:36:40.247083327Z 10:36:40,851 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - leader22: Updated server config: isVoting: true, peers: [PeerInfo [id=follower23, address=pekko://test/user/follower23, votingState=NON_VOTING]] 10:36:40,852 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader22: support() called 10:36:40,852 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader22: Moving last applied index from -1 to -1 10:36:40,852 AM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - leader22: Recovery completed in in 1.191 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:40,852 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader22: Recovery completed and cannot restore from snapshot 10:36:41,285 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader22 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.75:17355, Removed) 10:36:41,286 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader22 (Follower): Received ElectionTimeout - switching to Candidate 10:36:41,286 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader22 (Candidate): Election: Candidate has following voting peers: [] 10:36:41,289 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader22: Set currentTerm=3, votedFor=leader22 10:36:41,289 AM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - leader22 (Candidate): Starting new election term 3 10:36:41,289 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader22 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 10:36:41,289 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader22 (Candidate): Received ElectionTimeout 10:36:41,289 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader22 (Leader): Election: Leader has following peers: [follower23] 10:36:41,289 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:41,290 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader22 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 10:36:41,313 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader22: Persist data index=0 term=3 command=zero-1 10:36:41,313 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader22: Append log entry and persist JournaledLogEntry{index=0, term=3, command=zero-1, pending=true} 10:36:41,313 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader22: 1 completion(s) pending 10:36:41,313 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader22 (Leader): Replicate message: identifier: MockIdentifier{value=zero-1}, logIndex: 0, isSendImmediate: true 10:36:41,313 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader22 (Leader): Setting last applied to 0 10:36:41,313 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader22: Moving last applied index from -1 to 0 10:36:41,313 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader22: Applying state for log index 0 data zero-1 10:36:41,313 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader22: applyState called: zero-1 10:36:41,314 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:41,314 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader22: running 1 completion(s) 10:36:41,314 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader22: persist complete JournaledLogEntry{index=0, term=3, command=zero-1, pending=true} 10:36:41,314 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader22: Persist data index=1 term=3 command=one-1 10:36:41,314 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader22: Append log entry and persist JournaledLogEntry{index=1, term=3, command=one-1, pending=true} 10:36:41,314 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader22: 1 completion(s) pending 10:36:41,314 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader22 (Leader): Replicate message: identifier: MockIdentifier{value=one-1}, logIndex: 1, isSendImmediate: true 10:36:41,315 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader22 (Leader): Setting last applied to 1 10:36:41,315 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader22: Moving last applied index from 0 to 1 10:36:41,315 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader22: Applying state for log index 1 data one-1 10:36:41,315 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader22: applyState called: one-1 10:36:41,315 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader22 (Leader): Checking sendAppendEntries for follower follower23: active: false, followerNextIndex: -1, leaderLastIndex: 1, leaderSnapShotIndex: -1 10:36:41,315 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader22: running 1 completion(s) 10:36:41,315 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader22: persist complete JournaledLogEntry{index=1, term=3, command=one-1, pending=true} 10:36:41,406 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower23 (Follower): Term 3 in "AppendEntries{term=3, leaderId=leader22, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 10:36:41,409 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower23: Set currentTerm=3, votedFor=null 10:36:41,409 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower23: Lagging -3 entries behind leader leader22 and reached 2 (of expected -1) 10:36:41,409 AM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractLeader - leader22 (Leader): handleAppendEntriesReply: follower follower23 lastIndex 2 is ahead of our lastIndex 1 (snapshotIndex -1, snapshotTerm -1) - forcing install snaphot 10:36:41,409 AM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader22: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=1, lastAppliedTerm=3, lastIndex=1, lastTerm=3, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower23 10:36:41,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader22: last applied journal index prior to capture: 2 10:36:41,417 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader22: support() called 10:36:41,422 AM [leader22-%d504604] [DEBUG] RaftStorageCompleter - leader22: 1 completion(s) pending 10:36:41,422 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader22: running 1 completion(s) 10:36:41,422 AM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader22: Persising snapshot at EntryInfo[index=1, term=3]/EntryInfo[index=1, term=3] 10:36:41,422 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader22: support() called 10:36:41,423 AM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader22: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 10:36:41,423 AM [leader22-%d504605] [DEBUG] RaftStorage - leader22: starting snapshot writeout to /tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF439-193D5C88.tmp 10:36:41,424 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader22 (Leader): sendInstallSnapshot 10:36:41,424 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] LeaderInstallSnapshotState - leader22 (Leader): Snapshot 298 bytes, total chunks to send: 1 10:36:41,424 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] LeaderInstallSnapshotState - leader22 (Leader): Next chunk: total length=298, offset=0, size=298, hashCode=-1896618261 10:36:41,424 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader22 (Leader): next snapshot chunk size for follower follower23: 298 10:36:41,425 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader22 (Leader): InstallSnapshot sent to follower Vector(user, follower23), Chunk: 1/1 10:36:41,425 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower23 (Follower): handleInstallSnapshot: InstallSnapshot{term=3, leaderId=leader22, lastIncludedIndex=1, lastIncludedTerm=3, datasize=298, chunk=1/1, lastChunkHashCode=OptionalInt[-1], votingConfig=VotingConfig{serverInfo=[ServerInfo[peerId=follower23, isVoting=false], ServerInfo[peerId=leader22, isVoting=true]]}} 10:36:41,426 AM [leader22-%d504605] [DEBUG] RaftStorage - leader22: finished snapshot writeout to /tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF439-193D5C88.v1 10:36:41,426 AM [leader22-%d504605] [DEBUG] RaftStorage - leader22: selected SNAPSHOT_V1 to handle file /tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF438-0EBA313F.v1 10:36:41,426 AM [leader22-%d504605] [DEBUG] RaftStorage - leader22: skipping unrecognized file /tmp/junit-17223293662477380498/leader22/TermInfo.properties 10:36:41,426 AM [leader22-%d504605] [DEBUG] RaftStorage - leader22: skipping unrecognized file /tmp/junit-17223293662477380498/leader22/journal-v1-1.log 10:36:41,426 AM [leader22-%d504605] [DEBUG] RaftStorage - leader22: selected SNAPSHOT_V1 to handle file /tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF439-193D5C88.v1 10:36:41,426 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower23: Lagging -1 entries behind leader leader22 and reached 2 (of expected -1) 10:36:41,426 AM [leader22-%d504605] [DEBUG] RaftStorage - leader22: skipping unrecognized file /tmp/junit-17223293662477380498/leader22/metadata-v1-1.log 10:36:41,426 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotTracker - follower23 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 10:36:41,427 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower23 (Follower): Snapshot received from leader: leader22 10:36:41,427 AM [leader22-%d504605] [DEBUG] RaftStorage - leader22: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:40.247083327Z, file=/tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF438-0EBA313F.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:41,427 AM [leader22-%d504605] [DEBUG] RaftStorage - leader22: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:41.423451784Z, file=/tmp/junit-17223293662477380498/leader22/snapshot-0000000069AFF439-193D5C88.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:41,427 AM [leader22-%d504605] [DEBUG] RaftStorageCompleter - leader22: 1 completion(s) pending 10:36:41,427 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader22: running 1 completion(s) 10:36:41,427 AM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader22: snapshot is durable as of 2026-03-10T10:36:41.423451784Z 10:36:41,427 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - leader22: Snapshot success, discarding journal entries up to 2 10:36:41,427 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorSnapshotMessageSupport - leader22: SnapshotComplete received 10:36:41,428 AM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - follower23: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=298}} 10:36:41,428 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower23: support() called 10:36:41,430 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - follower23: Converted InstallSnapshot from leader: leader22 to state 10:36:41,431 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - follower23: last applied journal index prior to persisting applied snapshot: 3 10:36:41,431 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower23: support() called 10:36:41,431 AM [follower23-%d81144] [DEBUG] RaftStorage - follower23: starting snapshot writeout to /tmp/junit-17223293662477380498/follower23/snapshot-0000000069AFF439-19BA05EC.tmp 10:36:41,433 AM [follower23-%d81144] [DEBUG] RaftStorage - follower23: finished snapshot writeout to /tmp/junit-17223293662477380498/follower23/snapshot-0000000069AFF439-19BA05EC.v1 10:36:41,433 AM [follower23-%d81144] [DEBUG] RaftStorage - follower23: selected SNAPSHOT_V1 to handle file /tmp/junit-17223293662477380498/follower23/snapshot-0000000069AFF439-19BA05EC.v1 10:36:41,434 AM [follower23-%d81144] [DEBUG] RaftStorage - follower23: skipping unrecognized file /tmp/junit-17223293662477380498/follower23/TermInfo.properties 10:36:41,434 AM [follower23-%d81144] [DEBUG] RaftStorage - follower23: skipping unrecognized file /tmp/junit-17223293662477380498/follower23/journal-v1-1.log 10:36:41,434 AM [follower23-%d81144] [DEBUG] RaftStorage - follower23: skipping unrecognized file /tmp/junit-17223293662477380498/follower23/metadata-v1-1.log 10:36:41,434 AM [follower23-%d81144] [DEBUG] RaftStorage - follower23: selected SNAPSHOT_V1 to handle file /tmp/junit-17223293662477380498/follower23/snapshot-0000000069AFF438-0ECE43FF.v1 10:36:41,434 AM [follower23-%d81144] [DEBUG] RaftStorage - follower23: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:40.248398847Z, file=/tmp/junit-17223293662477380498/follower23/snapshot-0000000069AFF438-0ECE43FF.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:41,434 AM [follower23-%d81144] [DEBUG] RaftStorage - follower23: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:41.431621612Z, file=/tmp/junit-17223293662477380498/follower23/snapshot-0000000069AFF439-19BA05EC.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:41,434 AM [follower23-%d81144] [DEBUG] RaftStorageCompleter - follower23: 1 completion(s) pending 10:36:41,434 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower23: running 1 completion(s) 10:36:41,434 AM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - follower23: snapshot is durable as of 2026-03-10T10:36:41.431621612Z 10:36:41,435 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - follower23: Updated server config: isVoting: false, peers: [PeerInfo [id=leader22, address=pekko://test/user/leader22, votingState=VOTING]] 10:36:41,435 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower23: support() called 10:36:41,436 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower23: applySnapshot called 10:36:41,436 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower23 (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=3, followerId=follower23, chunkIndex=1, success=true} 10:36:41,437 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - follower23: Snapshot success, discarding journal entries up to 3 10:36:41,437 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorSnapshotMessageSupport - follower23: SnapshotComplete received 10:36:41,437 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader22 (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=3, followerId=follower23, chunkIndex=1, success=true} 10:36:41,437 AM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractLeader - leader22 (Leader): Snapshot successfully installed on follower follower23 (last chunk 1) - matchIndex set to 1, nextIndex set to 2 10:36:41,437 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader22: fakeSnapshot purging log to 0 for term 3 10:36:41,465 AM [main] [INFO] NonVotingFollowerIntegrationTest - testFollowerResyncWithLessLeaderLogEntriesAfterNonPersistentLeaderRestart ending 10:36:41,465 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower23-collector25#-2138329220] 10:36:41,478 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower23] 10:36:41,478 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - follower23: terminating thread VirtualThread[#126,follower23-writer-11]/waiting 10:36:41,478 AM [follower23-writer-11] [DEBUG] JournalWriteTask - follower23: received 1 commands after 1.221 s 10:36:41,479 AM [follower23-writer-11] [DEBUG] JournalWriteTask - follower23: completed 1 commands in 27.83 μs 10:36:41,479 AM [follower23-writer-11] [DEBUG] JournalWriteTask - follower23: journal writer stopped 10:36:41,479 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower23: journal closed 10:36:41,479 AM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - follower23: journal closed 10:36:41,479 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower23: stopped executor with 0 remaining tasks 10:36:41,479 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower23: stopped executor with 0 remaining tasks 10:36:41,480 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader22-collector26#-758470248] 10:36:41,480 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader22] 10:36:41,481 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader22-collector27#-1933889274] 10:36:41,482 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader22] 10:36:41,482 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - leader22: terminating thread VirtualThread[#128,leader22-writer-13]/waiting 10:36:41,482 AM [leader22-writer-13] [DEBUG] JournalWriteTask - leader22: received 1 commands after 631.4 ms 10:36:41,482 AM [leader22-writer-13] [DEBUG] JournalWriteTask - leader22: completed 1 commands in 24.78 μs 10:36:41,482 AM [leader22-writer-13] [DEBUG] JournalWriteTask - leader22: journal writer stopped 10:36:41,483 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader22: journal closed 10:36:41,483 AM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader22: journal closed 10:36:41,483 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: stopped executor with 0 remaining tasks 10:36:41,483 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader22: stopped executor with 0 remaining tasks 10:36:41,487 AM [main] [INFO] NonVotingFollowerIntegrationTest - testFollowerResyncWithEmptyLeaderLogAfterNonPersistentLeaderRestart starting 10:36:41,490 AM [main] [DEBUG] PropertiesTermInfoStore - leader28: Set currentTerm=1, votedFor=leader28 10:36:41,491 AM [main] [DEBUG] PropertiesTermInfoStore - follower29: Set currentTerm=1, votedFor=leader28 10:36:41,491 AM [main] [DEBUG] RaftStorage - leader28: starting snapshot writeout to /tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF439-1D51199A.tmp 10:36:41,495 AM [main] [DEBUG] RaftStorage - leader28: finished snapshot writeout to /tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF439-1D51199A.v1 10:36:41,495 AM [main] [DEBUG] RaftStorage - follower29: starting snapshot writeout to /tmp/junit-10477795132436081578/follower29/snapshot-0000000069AFF439-1D841B99.tmp 10:36:41,496 AM [main] [DEBUG] RaftStorage - follower29: finished snapshot writeout to /tmp/junit-10477795132436081578/follower29/snapshot-0000000069AFF439-1D841B99.v1 10:36:41,497 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/follower29] [WARN] [03/10/2026 10:36:41.498] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower29]: org.opendaylight.controller.cluster.common.actor.Monitor@2a5ccfc1 [INFO] [pekkoDeadLetter][03/10/2026 10:36:41.498] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower29] to Actor[pekko://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:41,498 AM [test-pekko.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower29] to Actor[pekko://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 10:36:41,499 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower29: Starting recovery with journal batch size 1000 10:36:41,499 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower29: started executor 10:36:41,499 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower29: started executor 10:36:41,500 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower29: metadata journal open: replayFrom=1 applyTo=0 10:36:41,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower29: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:41,501 AM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - follower29: journal open: applyTo=0 10:36:41,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower29: recovering to non-persistent 10:36:41,501 AM [follower29-writer-14] [DEBUG] JournalWriteTask - follower29: journal writer started 10:36:41,501 AM [follower29-writer-14] [DEBUG] JournalWriteTask - follower29: waiting to receive commands 10:36:41,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower29: Set currentTerm=1, votedFor=leader28 10:36:41,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower29: selected SNAPSHOT_V1 to handle file /tmp/junit-10477795132436081578/follower29/snapshot-0000000069AFF439-1D841B99.v1 10:36:41,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower29: skipping unrecognized file /tmp/junit-10477795132436081578/follower29/TermInfo.properties 10:36:41,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower29: skipping unrecognized file /tmp/junit-10477795132436081578/follower29/journal-v1-1.log 10:36:41,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower29: skipping unrecognized file /tmp/junit-10477795132436081578/follower29/metadata-v1-1.log 10:36:41,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower29: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:41.495197081Z, file=/tmp/junit-10477795132436081578/follower29/snapshot-0000000069AFF439-1D841B99.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:41,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Recovery - follower29: initializing from snapshot taken at 2026-03-10T10:36:41.495197081Z 10:36:41,502 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower29: Updated server config: isVoting: false, peers: [PeerInfo [id=leader28, address=pekko://test/user/leader28, votingState=VOTING]] 10:36:41,502 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower29: support() called 10:36:41,502 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower29: Moving last applied index from -1 to -1 10:36:41,502 AM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - follower29: Recovery completed in in 1.319 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:41,502 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower29: Recovery completed and cannot restore from snapshot 10:36:41,517 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/leader28] [WARN] [03/10/2026 10:36:41.517] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader28]: org.opendaylight.controller.cluster.common.actor.Monitor@54d4f4f7 10:36:41,518 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader28: Starting recovery with journal batch size 1000 10:36:41,518 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader28: started executor 10:36:41,518 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader28: started executor 10:36:41,519 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader28: metadata journal open: replayFrom=1 applyTo=0 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader28: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader28: journal open: applyTo=0 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader28: recovering to non-persistent 10:36:41,520 AM [leader28-writer-15] [DEBUG] JournalWriteTask - leader28: journal writer started 10:36:41,520 AM [leader28-writer-15] [DEBUG] JournalWriteTask - leader28: waiting to receive commands 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader28: Set currentTerm=1, votedFor=leader28 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader28: selected SNAPSHOT_V1 to handle file /tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF439-1D51199A.v1 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/TermInfo.properties 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/journal-v1-1.log 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/metadata-v1-1.log 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader28: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:41.491854234Z, file=/tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF439-1D51199A.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Recovery - leader28: initializing from snapshot taken at 2026-03-10T10:36:41.491854234Z 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - leader28: Updated server config: isVoting: true, peers: [PeerInfo [id=follower29, address=pekko://test/user/follower29, votingState=NON_VOTING]] 10:36:41,520 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader28: support() called 10:36:41,521 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader28: Moving last applied index from -1 to -1 10:36:41,521 AM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - leader28: Recovery completed in in 962.5 μs: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:41,521 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader28: Recovery completed and cannot restore from snapshot 10:36:42,25 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader28 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.75:17355, Removed) 10:36:42,25 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader28 (Follower): Received ElectionTimeout - switching to Candidate 10:36:42,25 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader28 (Candidate): Election: Candidate has following voting peers: [] 10:36:42,29 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader28: Set currentTerm=2, votedFor=leader28 10:36:42,29 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader28 (Candidate): Starting new election term 2 10:36:42,29 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActorBehavior - leader28 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 10:36:42,29 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader28 (Candidate): Received ElectionTimeout 10:36:42,30 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader28 (Leader): Election: Leader has following peers: [follower29] 10:36:42,30 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:42,30 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActorBehavior - leader28 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 10:36:42,30 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower29 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader28, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 10:36:42,33 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower29: Set currentTerm=2, votedFor=null 10:36:42,33 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower29: Last sync leader does not match current leader leader28, need to catch up to -1 10:36:42,34 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader28 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower29 updated: matchIndex: -1, nextIndex: 0 10:36:42,34 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/10/2026 10:36:42.034] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower29] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower29, initialSyncDone=false] 10:36:42,82 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader28: Persist data index=0 term=2 command=zero 10:36:42,82 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader28: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 10:36:42,82 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader28: 1 completion(s) pending 10:36:42,82 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 10:36:42,83 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader28 (Leader): Setting last applied to 0 10:36:42,83 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader28: Moving last applied index from -1 to 0 10:36:42,83 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader28: Applying state for log index 0 data zero 10:36:42,83 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader28: applyState called: zero 10:36:42,83 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:42,83 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): sendAppendEntries: 0 is present for follower follower29 10:36:42,83 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Sending AppendEntries to follower follower29: AppendEntries{term=2, leaderId=leader28, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 10:36:42,83 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader28: running 1 completion(s) 10:36:42,83 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader28: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} 10:36:42,83 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader28, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader28: Persist data index=1 term=2 command=one 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): Number of entries to be appended = 1 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader28: Append log entry and persist JournaledLogEntry{index=1, term=2, command=one, pending=true} 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader28: 1 completion(s) pending 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): Append entry to log zero 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - follower29: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=false} 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader28 (Leader): Setting last applied to 1 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader28: Moving last applied index from 0 to 1 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader28: Applying state for log index 1 data one 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader28: applyState called: one 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower29: 1 completion(s) pending 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): Log size is now 1 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): sendAppendEntries: 0 is present for follower follower29 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): Commit index set to 0 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Sending AppendEntries to follower follower29: AppendEntries{term=2, leaderId=leader28, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}, JournaledLogEntry{index=1, term=2, command=one, pending=true}]} 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower29, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower29: Lagging 0 entries behind leader leader28 and reached 0 (of expected -1) 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader28: running 1 completion(s) 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader28: persist complete JournaledLogEntry{index=1, term=2, command=one, pending=true} 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - follower29 (Follower): Setting last applied to 0 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - follower29: Moving last applied index from -1 to 0 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - follower29: Applying state for log index 0 data zero 10:36:42,84 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower29: applyState called: zero 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower29 updated: matchIndex: 0, nextIndex: 1 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - leader28: fakeSnapshot purging log to 0 for term 2 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: 0 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): sendAppendEntries: 1 is present for follower follower29 10:36:42,84 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Sending AppendEntries to follower follower29: AppendEntries{term=2, leaderId=leader28, prevLogIndex=0, prevLogTerm=2, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=one, pending=false}]} 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader28: Persist data index=2 term=2 command=two 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader28: Append log entry and persist JournaledLogEntry{index=2, term=2, command=two, pending=true} 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader28: 1 completion(s) pending 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Replicate message: identifier: MockIdentifier{value=two}, logIndex: 2, isSendImmediate: true 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader28 (Leader): Setting last applied to 2 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader28: Moving last applied index from 1 to 2 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader28: Applying state for log index 2 data two 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader28: applyState called: two 10:36:42,85 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower29: running 1 completion(s) 10:36:42,85 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - follower29: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: true, followerNextIndex: 1, leaderLastIndex: 2, leaderSnapShotIndex: 0 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): sendAppendEntries: 1 is present for follower follower29 10:36:42,85 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower29: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=2, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@3aa618f7} 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader28 (Leader): Sending AppendEntries to follower follower29: AppendEntries{term=2, leaderId=leader28, prevLogIndex=0, prevLogTerm=2, leaderCommit=2, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=one, pending=false}, JournaledLogEntry{index=2, term=2, command=two, pending=true}]} 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader28: running 1 completion(s) 10:36:42,85 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader28: persist complete JournaledLogEntry{index=2, term=2, command=two, pending=true} 10:36:42,85 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower29: synchronized callbacks after 236.6 μs 10:36:42,85 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader28, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}, JournaledLogEntry{index=1, term=2, command=one, pending=false}]} 10:36:42,85 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): Number of entries to be appended = 2 10:36:42,85 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): matchEntry JournaledLogEntry{index=0, term=2, command=zero, pending=false} is present: existingEntryTerm: 2 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): After cleanup, lastIndex: 0, entries to be added from: 1 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): Append entry to log one 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - follower29: Append log entry and persist JournaledLogEntry{index=1, term=2, command=one, pending=false} 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower29: 1 completion(s) pending 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): Log size is now 2 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): Commit index set to 1 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower29, logLastIndex=1, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower29: Lagging 0 entries behind leader leader28 and reached 1 (of expected -1) 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - follower29 (Follower): Setting last applied to 1 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - follower29: Moving last applied index from 0 to 1 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - follower29: Applying state for log index 1 data one 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower29: applyState called: one 10:36:42,86 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader28 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower29 updated: matchIndex: 1, nextIndex: 2 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower29: running 1 completion(s) 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - follower29: persist complete JournaledLogEntry{index=1, term=2, command=one, pending=false} 10:36:42,86 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader28: fakeSnapshot purging log to 1 for term 2 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower29: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=2, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@75b78216} 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower29: synchronized callbacks after 100.7 μs 10:36:42,86 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 1 10:36:42,86 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader28 (Leader): sendAppendEntries: 2 is present for follower follower29 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader28, prevLogIndex=0, prevLogTerm=2, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=one, pending=false}]} 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): Number of entries to be appended = 1 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): matchEntry JournaledLogEntry{index=1, term=2, command=one, pending=false} is present: existingEntryTerm: 2 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): After cleanup, lastIndex: 1, entries to be added from: 1 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): Log size is now 2 10:36:42,86 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader28 (Leader): Sending AppendEntries to follower follower29: AppendEntries{term=2, leaderId=leader28, prevLogIndex=1, prevLogTerm=2, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=2, command=two, pending=false}]} 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower29, logLastIndex=1, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower29: Lagging 0 entries behind leader leader28 and reached 1 (of expected -1) 10:36:42,86 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - follower29: fakeSnapshot purging log to 0 for term 2 10:36:42,87 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader28 (Leader): sendAppendEntries: 2 is present for follower follower29 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader28, prevLogIndex=0, prevLogTerm=2, leaderCommit=2, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=one, pending=false}, JournaledLogEntry{index=2, term=2, command=two, pending=false}]} 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): Number of entries to be appended = 2 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): matchEntry JournaledLogEntry{index=1, term=2, command=one, pending=false} is present: existingEntryTerm: 2 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): After cleanup, lastIndex: 1, entries to be added from: 1 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): Append entry to log two 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower29: Append log entry and persist JournaledLogEntry{index=2, term=2, command=two, pending=false} 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower29: 1 completion(s) pending 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): Log size is now 2 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): Commit index set to 2 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower29, logLastIndex=2, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower29: Lagging 0 entries behind leader leader28 and reached 2 (of expected -1) 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 2 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower29 (Follower): Setting last applied to 2 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower29: Moving last applied index from 1 to 2 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower29: Applying state for log index 2 data two 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower29: applyState called: two 10:36:42,87 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader28 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower29 updated: matchIndex: 2, nextIndex: 3 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower29: running 1 completion(s) 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower29: persist complete JournaledLogEntry{index=2, term=2, command=two, pending=false} 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower29: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=2, command=two, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@1150d1b6} 10:36:42,87 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 1 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower29: synchronized callbacks after 86.85 μs 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader28, prevLogIndex=1, prevLogTerm=2, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=2, command=two, pending=false}]} 10:36:42,87 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): Number of entries to be appended = 1 10:36:42,88 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): matchEntry JournaledLogEntry{index=2, term=2, command=two, pending=false} is present: existingEntryTerm: 2 10:36:42,88 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): After cleanup, lastIndex: 2, entries to be added from: 1 10:36:42,88 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): Log size is now 2 10:36:42,88 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower29 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower29, logLastIndex=2, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:42,88 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower29: Lagging 0 entries behind leader leader28 and reached 2 (of expected -1) 10:36:42,88 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - follower29: fakeSnapshot purging log to 1 for term 2 [WARN] [03/10/2026 10:36:42.088] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower29] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower29, initialSyncDone=true] 10:36:42,134 AM [main] [DEBUG] RaftStorage - leader28: selected SNAPSHOT_V1 to handle file /tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF439-1D51199A.v1 10:36:42,134 AM [main] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/TermInfo.properties 10:36:42,134 AM [main] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/journal-v1-1.log 10:36:42,134 AM [main] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/metadata-v1-1.log 10:36:42,134 AM [main] [DEBUG] RaftStorage - leader28: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:41.491854234Z, file=/tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF439-1D51199A.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:42,146 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - leader28: terminating thread VirtualThread[#133,leader28-writer-15]/waiting 10:36:42,146 AM [leader28-writer-15] [DEBUG] JournalWriteTask - leader28: received 1 commands after 626.7 ms 10:36:42,147 AM [leader28-writer-15] [DEBUG] JournalWriteTask - leader28: completed 1 commands in 32.77 μs 10:36:42,147 AM [leader28-writer-15] [DEBUG] JournalWriteTask - leader28: journal writer stopped 10:36:42,147 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader28: journal closed 10:36:42,147 AM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader28: journal closed 10:36:42,147 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: stopped executor with 0 remaining tasks 10:36:42,147 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: stopped executor with 0 remaining tasks 10:36:42,149 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/leader28] [WARN] [03/10/2026 10:36:42.150] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader28]: org.opendaylight.controller.cluster.common.actor.Monitor@7fdc5df7 10:36:42,151 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader28: Starting recovery with journal batch size 1000 10:36:42,151 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: started executor 10:36:42,151 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: started executor 10:36:42,151 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader28: metadata journal open: replayFrom=1 applyTo=0 10:36:42,152 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader28: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:42,152 AM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader28: journal open: applyTo=0 10:36:42,152 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader28: recovering to non-persistent 10:36:42,152 AM [leader28-writer-16] [DEBUG] JournalWriteTask - leader28: journal writer started 10:36:42,152 AM [leader28-writer-16] [DEBUG] JournalWriteTask - leader28: waiting to receive commands 10:36:42,152 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader28: Set currentTerm=2, votedFor=leader28 10:36:42,153 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: selected SNAPSHOT_V1 to handle file /tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF439-1D51199A.v1 10:36:42,153 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/TermInfo.properties 10:36:42,153 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/journal-v1-1.log 10:36:42,153 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/metadata-v1-1.log 10:36:42,153 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:41.491854234Z, file=/tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF439-1D51199A.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:42,153 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Recovery - leader28: initializing from snapshot taken at 2026-03-10T10:36:41.491854234Z 10:36:42,153 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - leader28: Updated server config: isVoting: true, peers: [PeerInfo [id=follower29, address=pekko://test/user/follower29, votingState=NON_VOTING]] 10:36:42,153 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - leader28: support() called 10:36:42,154 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader28: Moving last applied index from -1 to -1 10:36:42,154 AM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader28: Recovery completed in in 1.409 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:42,154 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader28: Recovery completed and cannot restore from snapshot 10:36:42,595 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - leader28 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.75:17355, Removed) 10:36:42,595 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - leader28 (Follower): Received ElectionTimeout - switching to Candidate 10:36:42,596 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader28 (Candidate): Election: Candidate has following voting peers: [] 10:36:42,599 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader28: Set currentTerm=3, votedFor=leader28 10:36:42,599 AM [test-pekko.actor.default-dispatcher-13] [INFO] Candidate - leader28 (Candidate): Starting new election term 3 10:36:42,599 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - leader28 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 10:36:42,599 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader28 (Candidate): Received ElectionTimeout 10:36:42,599 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader28 (Leader): Election: Leader has following peers: [follower29] 10:36:42,600 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:42,600 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - leader28 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 10:36:42,600 AM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower29 (Follower): Term 3 in "AppendEntries{term=3, leaderId=leader28, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 10:36:42,603 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower29: Set currentTerm=3, votedFor=null 10:36:42,603 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower29: Lagging -3 entries behind leader leader28 and reached 2 (of expected -1) 10:36:42,604 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractLeader - leader28 (Leader): handleAppendEntriesReply: follower follower29 lastIndex 2 is ahead of our lastIndex -1 (snapshotIndex -1, snapshotTerm -1) - forcing install snaphot 10:36:42,604 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader28: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 10:36:42,604 AM [test-pekko.actor.default-dispatcher-14] [INFO] SnapshotManager - leader28: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower29 10:36:42,604 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - leader28: last applied journal index prior to capture: 0 10:36:42,604 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader28: support() called 10:36:42,606 AM [leader28-%d634146] [DEBUG] RaftStorageCompleter - leader28: 1 completion(s) pending 10:36:42,606 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader28: running 1 completion(s) 10:36:42,606 AM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader28: Persising snapshot at EntryInfo[index=-1, term=-1]/EntryInfo[index=-1, term=-1] 10:36:42,606 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - leader28: support() called 10:36:42,606 AM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader28: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 10:36:42,606 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader28 (Leader): sendInstallSnapshot 10:36:42,607 AM [leader28-%d634147] [DEBUG] RaftStorage - leader28: starting snapshot writeout to /tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF43A-242CA0DA.tmp 10:36:42,607 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] LeaderInstallSnapshotState - leader28 (Leader): Snapshot 157 bytes, total chunks to send: 1 10:36:42,607 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] LeaderInstallSnapshotState - leader28 (Leader): Next chunk: total length=157, offset=0, size=157, hashCode=1546036366 10:36:42,607 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader28 (Leader): next snapshot chunk size for follower follower29: 157 10:36:42,607 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): handleInstallSnapshot: InstallSnapshot{term=3, leaderId=leader28, lastIncludedIndex=-1, lastIncludedTerm=-1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1], votingConfig=VotingConfig{serverInfo=[ServerInfo[peerId=follower29, isVoting=false], ServerInfo[peerId=leader28, isVoting=true]]}} 10:36:42,607 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower29: Lagging -3 entries behind leader leader28 and reached 2 (of expected -1) 10:36:42,607 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotTracker - follower29 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 10:36:42,607 AM [test-pekko.actor.default-dispatcher-14] [INFO] Follower - follower29 (Follower): Snapshot received from leader: leader28 10:36:42,607 AM [test-pekko.actor.default-dispatcher-14] [INFO] SnapshotManager - follower29: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=157}} 10:36:42,607 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower29: support() called 10:36:42,608 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader28 (Leader): InstallSnapshot sent to follower Vector(user, follower29), Chunk: 1/1 10:36:42,608 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - follower29: Converted InstallSnapshot from leader: leader28 to state 10:36:42,608 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - follower29: last applied journal index prior to persisting applied snapshot: 3 10:36:42,608 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower29: support() called 10:36:42,609 AM [follower29-%d763353] [DEBUG] RaftStorage - follower29: starting snapshot writeout to /tmp/junit-10477795132436081578/follower29/snapshot-0000000069AFF43A-2448FE4E.tmp 10:36:42,609 AM [leader28-%d634147] [DEBUG] RaftStorage - leader28: finished snapshot writeout to /tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF43A-242CA0DA.v1 10:36:42,609 AM [leader28-%d634147] [DEBUG] RaftStorage - leader28: selected SNAPSHOT_V1 to handle file /tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF439-1D51199A.v1 10:36:42,609 AM [leader28-%d634147] [DEBUG] RaftStorage - leader28: selected SNAPSHOT_V1 to handle file /tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF43A-242CA0DA.v1 10:36:42,609 AM [leader28-%d634147] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/TermInfo.properties 10:36:42,609 AM [leader28-%d634147] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/journal-v1-1.log 10:36:42,609 AM [leader28-%d634147] [DEBUG] RaftStorage - leader28: skipping unrecognized file /tmp/junit-10477795132436081578/leader28/metadata-v1-1.log 10:36:42,610 AM [leader28-%d634147] [DEBUG] RaftStorage - leader28: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:41.491854234Z, file=/tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF439-1D51199A.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:42,610 AM [leader28-%d634147] [DEBUG] RaftStorage - leader28: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:42.606904538Z, file=/tmp/junit-10477795132436081578/leader28/snapshot-0000000069AFF43A-242CA0DA.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:42,610 AM [leader28-%d634147] [DEBUG] RaftStorageCompleter - leader28: 1 completion(s) pending 10:36:42,610 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader28: running 1 completion(s) 10:36:42,610 AM [test-pekko.actor.default-dispatcher-14] [INFO] SnapshotManager - leader28: snapshot is durable as of 2026-03-10T10:36:42.606904538Z 10:36:42,610 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - leader28: Snapshot success, discarding journal entries up to 0 10:36:42,610 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorSnapshotMessageSupport - leader28: SnapshotComplete received 10:36:42,611 AM [follower29-%d763353] [DEBUG] RaftStorage - follower29: finished snapshot writeout to /tmp/junit-10477795132436081578/follower29/snapshot-0000000069AFF43A-2448FE4E.v1 10:36:42,611 AM [follower29-%d763353] [DEBUG] RaftStorage - follower29: selected SNAPSHOT_V1 to handle file /tmp/junit-10477795132436081578/follower29/snapshot-0000000069AFF43A-2448FE4E.v1 10:36:42,611 AM [follower29-%d763353] [DEBUG] RaftStorage - follower29: selected SNAPSHOT_V1 to handle file /tmp/junit-10477795132436081578/follower29/snapshot-0000000069AFF439-1D841B99.v1 10:36:42,611 AM [follower29-%d763353] [DEBUG] RaftStorage - follower29: skipping unrecognized file /tmp/junit-10477795132436081578/follower29/TermInfo.properties 10:36:42,611 AM [follower29-%d763353] [DEBUG] RaftStorage - follower29: skipping unrecognized file /tmp/junit-10477795132436081578/follower29/journal-v1-1.log 10:36:42,611 AM [follower29-%d763353] [DEBUG] RaftStorage - follower29: skipping unrecognized file /tmp/junit-10477795132436081578/follower29/metadata-v1-1.log 10:36:42,612 AM [follower29-%d763353] [DEBUG] RaftStorage - follower29: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:41.495197081Z, file=/tmp/junit-10477795132436081578/follower29/snapshot-0000000069AFF439-1D841B99.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:42,612 AM [follower29-%d763353] [DEBUG] RaftStorage - follower29: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:42.608763470Z, file=/tmp/junit-10477795132436081578/follower29/snapshot-0000000069AFF43A-2448FE4E.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:42,612 AM [follower29-%d763353] [DEBUG] RaftStorageCompleter - follower29: 1 completion(s) pending 10:36:42,612 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower29: running 1 completion(s) 10:36:42,612 AM [test-pekko.actor.default-dispatcher-14] [INFO] SnapshotManager - follower29: snapshot is durable as of 2026-03-10T10:36:42.608763470Z 10:36:42,612 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PeerInfos - follower29: Updated server config: isVoting: false, peers: [PeerInfo [id=leader28, address=pekko://test/user/leader28, votingState=VOTING]] 10:36:42,612 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower29: support() called 10:36:42,613 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower29: applySnapshot called 10:36:42,613 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower29 (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=3, followerId=follower29, chunkIndex=1, success=true} 10:36:42,613 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - follower29: Snapshot success, discarding journal entries up to 3 10:36:42,613 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorSnapshotMessageSupport - follower29: SnapshotComplete received 10:36:42,613 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader28 (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=3, followerId=follower29, chunkIndex=1, success=true} 10:36:42,613 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractLeader - leader28 (Leader): Snapshot successfully installed on follower follower29 (last chunk 1) - matchIndex set to -1, nextIndex set to 0 10:36:42,663 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader28: Persist data index=0 term=3 command=zero-1 10:36:42,663 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader28: Append log entry and persist JournaledLogEntry{index=0, term=3, command=zero-1, pending=true} 10:36:42,663 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader28: 1 completion(s) pending 10:36:42,663 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader28 (Leader): Replicate message: identifier: MockIdentifier{value=zero-1}, logIndex: 0, isSendImmediate: true 10:36:42,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader28 (Leader): Setting last applied to 0 10:36:42,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader28: Moving last applied index from -1 to 0 10:36:42,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader28: Applying state for log index 0 data zero-1 10:36:42,664 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader28: applyState called: zero-1 10:36:42,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:42,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader28 (Leader): sendAppendEntries: 0 is present for follower follower29 10:36:42,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader28 (Leader): Sending AppendEntries to follower follower29: AppendEntries{term=3, leaderId=leader28, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=3, command=zero-1, pending=true}]} 10:36:42,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader28: running 1 completion(s) 10:36:42,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader28: persist complete JournaledLogEntry{index=0, term=3, command=zero-1, pending=true} 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower29 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=leader28, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=3, command=zero-1, pending=false}]} 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower29 (Follower): Number of entries to be appended = 1 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower29 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower29 (Follower): Append entry to log zero-1 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower29: Append log entry and persist JournaledLogEntry{index=0, term=3, command=zero-1, pending=false} 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower29: 1 completion(s) pending 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower29 (Follower): Log size is now 1 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower29 (Follower): Commit index set to 0 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower29 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower29, logLastIndex=0, logLastTerm=3, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower29: Lagging 0 entries behind leader leader28 and reached 0 (of expected -1) 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower29 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower29 (Follower): Setting last applied to 0 10:36:42,665 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader28 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower29 updated: matchIndex: 0, nextIndex: 1 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower29: Moving last applied index from -1 to 0 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower29: Applying state for log index 0 data zero-1 10:36:42,665 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower29: applyState called: zero-1 10:36:42,665 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader28 (Leader): Checking sendAppendEntries for follower follower29: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:42,666 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower29: running 1 completion(s) 10:36:42,666 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower29: persist complete JournaledLogEntry{index=0, term=3, command=zero-1, pending=false} 10:36:42,666 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower29: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=3, command=zero-1, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@6301d6ef} 10:36:42,666 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower29: synchronized callbacks after 115.4 μs 10:36:42,714 AM [main] [INFO] NonVotingFollowerIntegrationTest - testFollowerResyncWithEmptyLeaderLogAfterNonPersistentLeaderRestart ending 10:36:42,715 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower29-collector31#520195506] 10:36:42,727 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower29] 10:36:42,728 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - follower29: terminating thread VirtualThread[#132,follower29-writer-14]/waiting 10:36:42,729 AM [follower29-writer-14] [DEBUG] JournalWriteTask - follower29: received 1 commands after 1.228 s 10:36:42,729 AM [follower29-writer-14] [DEBUG] JournalWriteTask - follower29: completed 1 commands in 41.01 μs 10:36:42,729 AM [follower29-writer-14] [DEBUG] JournalWriteTask - follower29: journal writer stopped 10:36:42,730 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower29: journal closed 10:36:42,730 AM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - follower29: journal closed 10:36:42,730 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower29: stopped executor with 0 remaining tasks 10:36:42,730 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower29: stopped executor with 0 remaining tasks 10:36:42,731 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader28-collector32#958428062] 10:36:42,732 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader28] 10:36:42,733 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader28-collector33#2070811949] 10:36:42,733 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader28] 10:36:42,733 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - leader28: terminating thread VirtualThread[#134,leader28-writer-16]/waiting 10:36:42,734 AM [leader28-writer-16] [DEBUG] JournalWriteTask - leader28: received 1 commands after 581.4 ms 10:36:42,734 AM [leader28-writer-16] [DEBUG] JournalWriteTask - leader28: completed 1 commands in 26.51 μs 10:36:42,734 AM [leader28-writer-16] [DEBUG] JournalWriteTask - leader28: journal writer stopped 10:36:42,734 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader28: journal closed 10:36:42,734 AM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader28: journal closed 10:36:42,734 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: stopped executor with 0 remaining tasks 10:36:42,734 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader28: stopped executor with 0 remaining tasks 10:36:42,738 AM [main] [INFO] NonVotingFollowerIntegrationTest - testFollowerResyncWithOneMoreLeaderLogEntryAfterNonPersistentLeaderRestart starting 10:36:42,740 AM [main] [DEBUG] PropertiesTermInfoStore - leader34: Set currentTerm=1, votedFor=leader34 10:36:42,742 AM [main] [DEBUG] PropertiesTermInfoStore - follower35: Set currentTerm=1, votedFor=leader34 10:36:42,742 AM [main] [DEBUG] RaftStorage - leader34: starting snapshot writeout to /tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43A-2C3DA5A5.tmp 10:36:42,743 AM [main] [DEBUG] RaftStorage - leader34: finished snapshot writeout to /tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43A-2C3DA5A5.v1 10:36:42,743 AM [main] [DEBUG] RaftStorage - follower35: starting snapshot writeout to /tmp/junit-13059131338678159093/follower35/snapshot-0000000069AFF43A-2C538D14.tmp 10:36:42,745 AM [main] [DEBUG] RaftStorage - follower35: finished snapshot writeout to /tmp/junit-13059131338678159093/follower35/snapshot-0000000069AFF43A-2C538D14.v1 10:36:42,746 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/follower35] [WARN] [03/10/2026 10:36:42.747] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower35]: org.opendaylight.controller.cluster.common.actor.Monitor@660d30d7 10:36:42,748 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - follower35: Starting recovery with journal batch size 1000 10:36:42,748 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower35: started executor 10:36:42,748 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower35: started executor 10:36:42,749 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower35: metadata journal open: replayFrom=1 applyTo=0 10:36:42,750 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower35: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:42,750 AM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - follower35: journal open: applyTo=0 10:36:42,750 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower35: recovering to non-persistent 10:36:42,750 AM [follower35-writer-17] [DEBUG] JournalWriteTask - follower35: journal writer started 10:36:42,750 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower35: Set currentTerm=1, votedFor=leader34 10:36:42,750 AM [follower35-writer-17] [DEBUG] JournalWriteTask - follower35: waiting to receive commands 10:36:42,750 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower35: skipping unrecognized file /tmp/junit-13059131338678159093/follower35/TermInfo.properties 10:36:42,750 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower35: skipping unrecognized file /tmp/junit-13059131338678159093/follower35/journal-v1-1.log 10:36:42,750 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower35: skipping unrecognized file /tmp/junit-13059131338678159093/follower35/metadata-v1-1.log 10:36:42,750 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower35: selected SNAPSHOT_V1 to handle file /tmp/junit-13059131338678159093/follower35/snapshot-0000000069AFF43A-2C538D14.v1 10:36:42,751 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower35: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:42.743673108Z, file=/tmp/junit-13059131338678159093/follower35/snapshot-0000000069AFF43A-2C538D14.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:42,751 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Recovery - follower35: initializing from snapshot taken at 2026-03-10T10:36:42.743673108Z 10:36:42,751 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - follower35: Updated server config: isVoting: false, peers: [PeerInfo [id=leader34, address=pekko://test/user/leader34, votingState=VOTING]] 10:36:42,751 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower35: support() called 10:36:42,751 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower35: Moving last applied index from -1 to -1 10:36:42,752 AM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - follower35: Recovery completed in in 1.429 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:42,752 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower35: Recovery completed and cannot restore from snapshot 10:36:42,754 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/leader34] [WARN] [03/10/2026 10:36:42.754] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader34]: org.opendaylight.controller.cluster.common.actor.Monitor@7b23fa04 10:36:42,755 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader34: Starting recovery with journal batch size 1000 10:36:42,756 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader34: started executor 10:36:42,756 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader34: started executor 10:36:42,756 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader34: metadata journal open: replayFrom=1 applyTo=0 10:36:42,757 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader34: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:42,757 AM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader34: journal open: applyTo=0 10:36:42,757 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader34: recovering to non-persistent 10:36:42,757 AM [leader34-writer-18] [DEBUG] JournalWriteTask - leader34: journal writer started 10:36:42,757 AM [leader34-writer-18] [DEBUG] JournalWriteTask - leader34: waiting to receive commands 10:36:42,757 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader34: Set currentTerm=1, votedFor=leader34 10:36:42,757 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader34: selected SNAPSHOT_V1 to handle file /tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43A-2C3DA5A5.v1 10:36:42,758 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader34: skipping unrecognized file /tmp/junit-13059131338678159093/leader34/TermInfo.properties 10:36:42,758 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader34: skipping unrecognized file /tmp/junit-13059131338678159093/leader34/journal-v1-1.log 10:36:42,758 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader34: skipping unrecognized file /tmp/junit-13059131338678159093/leader34/metadata-v1-1.log 10:36:42,758 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader34: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:42.742237605Z, file=/tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43A-2C3DA5A5.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:42,758 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Recovery - leader34: initializing from snapshot taken at 2026-03-10T10:36:42.742237605Z 10:36:42,758 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - leader34: Updated server config: isVoting: true, peers: [PeerInfo [id=follower35, address=pekko://test/user/follower35, votingState=NON_VOTING]] 10:36:42,758 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - leader34: support() called 10:36:42,758 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader34: Moving last applied index from -1 to -1 10:36:42,758 AM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader34: Recovery completed in in 1.108 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:42,758 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader34: Recovery completed and cannot restore from snapshot 10:36:43,235 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader34 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.75:17355, Removed) 10:36:43,235 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader34 (Follower): Received ElectionTimeout - switching to Candidate 10:36:43,235 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader34 (Candidate): Election: Candidate has following voting peers: [] 10:36:43,238 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader34: Set currentTerm=2, votedFor=leader34 10:36:43,238 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader34 (Candidate): Starting new election term 2 10:36:43,238 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActorBehavior - leader34 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 10:36:43,238 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader34 (Candidate): Received ElectionTimeout 10:36:43,238 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Election: Leader has following peers: [follower35] 10:36:43,239 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:43,239 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActorBehavior - leader34 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 10:36:43,239 AM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower35 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader34, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 10:36:43,242 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower35: Set currentTerm=2, votedFor=null 10:36:43,242 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower35: Last sync leader does not match current leader leader34, need to catch up to -1 10:36:43,242 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader34 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower35 updated: matchIndex: -1, nextIndex: 0 10:36:43,243 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/10/2026 10:36:43.243] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower35] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower35, initialSyncDone=false] 10:36:43,268 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader34: Persist data index=0 term=2 command=zero 10:36:43,268 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader34: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 10:36:43,268 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader34: 1 completion(s) pending 10:36:43,268 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 10:36:43,268 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader34 (Leader): Setting last applied to 0 10:36:43,268 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader34: Moving last applied index from -1 to 0 10:36:43,268 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader34: Applying state for log index 0 data zero 10:36:43,268 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader34: applyState called: zero 10:36:43,269 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:43,269 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): sendAppendEntries: 0 is present for follower follower35 10:36:43,269 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Sending AppendEntries to follower follower35: AppendEntries{term=2, leaderId=leader34, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 10:36:43,269 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower35 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader34, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower35 (Follower): Number of entries to be appended = 1 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower35 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower35 (Follower): Append entry to log zero 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower35: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower35: 1 completion(s) pending 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower35 (Follower): Log size is now 1 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower35 (Follower): Commit index set to 0 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower35 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower35, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower35: Lagging 0 entries behind leader leader34 and reached 0 (of expected -1) 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower35 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower35 (Follower): Setting last applied to 0 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower35: Moving last applied index from -1 to 0 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower35: Applying state for log index 0 data zero 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower35: applyState called: zero 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower35: running 1 completion(s) 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower35: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower35: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=2, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@5632c46c} 10:36:43,270 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower35: synchronized callbacks after 158.4 μs 10:36:43,271 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader34: running 1 completion(s) 10:36:43,271 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader34: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} [WARN] [03/10/2026 10:36:43.271] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower35] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower35, initialSyncDone=true] 10:36:43,271 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower35 updated: matchIndex: 0, nextIndex: 1 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader34: Persist data index=1 term=2 command=one 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader34: Append log entry and persist JournaledLogEntry{index=1, term=2, command=one, pending=true} 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader34: 1 completion(s) pending 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader34 (Leader): Setting last applied to 1 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader34: Moving last applied index from 0 to 1 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader34: Applying state for log index 1 data one 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader34: applyState called: one 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 10:36:43,272 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): sendAppendEntries: 1 is present for follower follower35 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Sending AppendEntries to follower follower35: AppendEntries{term=2, leaderId=leader34, prevLogIndex=0, prevLogTerm=2, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=one, pending=true}]} 10:36:43,273 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader34: running 1 completion(s) 10:36:43,273 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader34: persist complete JournaledLogEntry{index=1, term=2, command=one, pending=true} 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader34, prevLogIndex=0, prevLogTerm=2, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=one, pending=false}]} 10:36:43,273 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader34: fakeSnapshot purging log to 0 for term 2 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): Number of entries to be appended = 1 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): Append entry to log one 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - follower35: Append log entry and persist JournaledLogEntry{index=1, term=2, command=one, pending=false} 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower35: 1 completion(s) pending 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): Log size is now 2 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): Commit index set to 1 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower35, logLastIndex=1, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower35: Lagging 0 entries behind leader leader34 and reached 1 (of expected -1) 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - follower35 (Follower): Setting last applied to 1 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - follower35: Moving last applied index from 0 to 1 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - follower35: Applying state for log index 1 data one 10:36:43,273 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower35: applyState called: one 10:36:43,273 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader34 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower35 updated: matchIndex: 1, nextIndex: 2 10:36:43,274 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower35: running 1 completion(s) 10:36:43,274 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 10:36:43,274 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - follower35: persist complete JournaledLogEntry{index=1, term=2, command=one, pending=false} 10:36:43,274 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower35: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=2, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@3c4106ec} 10:36:43,274 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower35: synchronized callbacks after 299.9 μs 10:36:43,330 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EnabledRaftStorage - leader34: terminating thread VirtualThread[#139,leader34-writer-18]/waiting 10:36:43,331 AM [leader34-writer-18] [DEBUG] JournalWriteTask - leader34: received 1 commands after 573.4 ms 10:36:43,331 AM [leader34-writer-18] [DEBUG] JournalWriteTask - leader34: completed 1 commands in 64.55 μs 10:36:43,331 AM [leader34-writer-18] [DEBUG] JournalWriteTask - leader34: journal writer stopped 10:36:43,332 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader34: journal closed 10:36:43,332 AM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - leader34: journal closed 10:36:43,332 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: stopped executor with 0 remaining tasks 10:36:43,332 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: stopped executor with 0 remaining tasks 10:36:43,334 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/leader34] [WARN] [03/10/2026 10:36:43.336] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader34]: org.opendaylight.controller.cluster.common.actor.Monitor@11591347 10:36:43,336 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - leader34: Starting recovery with journal batch size 1000 10:36:43,336 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: started executor 10:36:43,336 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: started executor 10:36:43,337 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader34: metadata journal open: replayFrom=1 applyTo=0 10:36:43,338 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader34: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:43,338 AM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - leader34: journal open: applyTo=0 10:36:43,338 AM [leader34-writer-19] [DEBUG] JournalWriteTask - leader34: journal writer started 10:36:43,338 AM [leader34-writer-19] [DEBUG] JournalWriteTask - leader34: waiting to receive commands 10:36:43,338 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader34: recovering to non-persistent 10:36:43,338 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader34: Set currentTerm=2, votedFor=leader34 10:36:43,338 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: selected SNAPSHOT_V1 to handle file /tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43A-2C3DA5A5.v1 10:36:43,338 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: skipping unrecognized file /tmp/junit-13059131338678159093/leader34/TermInfo.properties 10:36:43,338 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: skipping unrecognized file /tmp/junit-13059131338678159093/leader34/journal-v1-1.log 10:36:43,338 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: skipping unrecognized file /tmp/junit-13059131338678159093/leader34/metadata-v1-1.log 10:36:43,338 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:42.742237605Z, file=/tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43A-2C3DA5A5.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:43,339 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Recovery - leader34: initializing from snapshot taken at 2026-03-10T10:36:42.742237605Z 10:36:43,339 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - leader34: Updated server config: isVoting: true, peers: [PeerInfo [id=follower35, address=pekko://test/user/follower35, votingState=NON_VOTING]] 10:36:43,339 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader34: support() called 10:36:43,339 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader34: Moving last applied index from -1 to -1 10:36:43,339 AM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - leader34: Recovery completed in in 1.173 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:43,339 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader34: Recovery completed and cannot restore from snapshot 10:36:43,815 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - leader34 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.75:17355, Removed) 10:36:43,815 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - leader34 (Follower): Received ElectionTimeout - switching to Candidate 10:36:43,815 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader34 (Candidate): Election: Candidate has following voting peers: [] 10:36:43,818 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader34: Set currentTerm=3, votedFor=leader34 10:36:43,818 AM [test-pekko.actor.default-dispatcher-11] [INFO] Candidate - leader34 (Candidate): Starting new election term 3 10:36:43,818 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader34 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 10:36:43,819 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader34 (Candidate): Received ElectionTimeout 10:36:43,819 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader34 (Leader): Election: Leader has following peers: [follower35] 10:36:43,819 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:43,820 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader34 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 10:36:43,851 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader34: Persist data index=0 term=3 command=zero-1 10:36:43,851 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader34: Append log entry and persist JournaledLogEntry{index=0, term=3, command=zero-1, pending=true} 10:36:43,852 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader34: 1 completion(s) pending 10:36:43,852 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Replicate message: identifier: MockIdentifier{value=zero-1}, logIndex: 0, isSendImmediate: true 10:36:43,852 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader34 (Leader): Setting last applied to 0 10:36:43,852 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader34: Moving last applied index from -1 to 0 10:36:43,852 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader34: Applying state for log index 0 data zero-1 10:36:43,852 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader34: applyState called: zero-1 10:36:43,852 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader34: running 1 completion(s) 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader34: persist complete JournaledLogEntry{index=0, term=3, command=zero-1, pending=true} 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader34: Persist data index=1 term=3 command=one-1 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader34: Append log entry and persist JournaledLogEntry{index=1, term=3, command=one-1, pending=true} 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader34: 1 completion(s) pending 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Replicate message: identifier: MockIdentifier{value=one-1}, logIndex: 1, isSendImmediate: true 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader34 (Leader): Setting last applied to 1 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader34: Moving last applied index from 0 to 1 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader34: Applying state for log index 1 data one-1 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader34: applyState called: one-1 10:36:43,853 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: false, followerNextIndex: -1, leaderLastIndex: 1, leaderSnapShotIndex: -1 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader34: running 1 completion(s) 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader34: persist complete JournaledLogEntry{index=1, term=3, command=one-1, pending=true} 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader34: Persist data index=2 term=3 command=two-1 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader34: Append log entry and persist JournaledLogEntry{index=2, term=3, command=two-1, pending=true} 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader34: 1 completion(s) pending 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Replicate message: identifier: MockIdentifier{value=two-1}, logIndex: 2, isSendImmediate: true 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader34 (Leader): Setting last applied to 2 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader34: Moving last applied index from 1 to 2 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader34: Applying state for log index 2 data two-1 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader34: applyState called: two-1 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: false, followerNextIndex: -1, leaderLastIndex: 2, leaderSnapShotIndex: -1 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader34: running 1 completion(s) 10:36:43,854 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader34: persist complete JournaledLogEntry{index=2, term=3, command=two-1, pending=true} 10:36:43,936 AM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower35 (Follower): Term 3 in "AppendEntries{term=3, leaderId=leader34, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 10:36:43,939 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower35: Set currentTerm=3, votedFor=null 10:36:43,939 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower35: Lagging -2 entries behind leader leader34 and reached 1 (of expected -1) 10:36:43,939 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractLeader - leader34 (Leader): handleAppendEntriesReply: follower follower35 last log term 2 for index 1 conflicts with the leader's 3 - set the follower's next index to 0 10:36:43,940 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Checking sendAppendEntries for follower follower35: active: true, followerNextIndex: 0, leaderLastIndex: 2, leaderSnapShotIndex: -1 10:36:43,940 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): sendAppendEntries: 0 is present for follower follower35 10:36:43,940 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader34 (Leader): Sending AppendEntries to follower follower35: AppendEntries{term=3, leaderId=leader34, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=2, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=3, command=zero-1, pending=false}, JournaledLogEntry{index=1, term=3, command=one-1, pending=false}, JournaledLogEntry{index=2, term=3, command=two-1, pending=false}]} 10:36:43,940 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=leader34, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=2, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=3, command=zero-1, pending=false}, JournaledLogEntry{index=1, term=3, command=one-1, pending=false}, JournaledLogEntry{index=2, term=3, command=two-1, pending=false}]} 10:36:43,940 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): Number of entries to be appended = 3 10:36:43,940 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): matchEntry JournaledLogEntry{index=0, term=3, command=zero-1, pending=false} is present: existingEntryTerm: 2 10:36:43,940 AM [test-pekko.actor.default-dispatcher-14] [INFO] Follower - follower35 (Follower): Removing entries from log starting at 0, commitIndex: 1, lastApplied: 1 10:36:43,940 AM [test-pekko.actor.default-dispatcher-14] [INFO] Follower - follower35 (Follower): Could not remove entries - sending reply to force snapshot 10:36:43,940 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower35: Lagging 1 entries behind leader leader34 and reached 1 (of expected -1) 10:36:43,941 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractLeader - leader34 (Leader): handleAppendEntriesReply - received unsuccessful reply: AppendEntriesReply{term=3, success=false, followerId=follower35, logLastIndex=1, logLastTerm=2, forceInstallSnapshot=true, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, leader snapshotIndex: -1, snapshotTerm: -1, replicatedToAllIndex: -1 10:36:43,941 AM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - leader34: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=2, lastAppliedTerm=3, lastIndex=2, lastTerm=3, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower35 10:36:43,941 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader34: last applied journal index prior to capture: 3 10:36:43,941 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader34: support() called 10:36:43,942 AM [leader34-%d689804] [DEBUG] RaftStorageCompleter - leader34: 1 completion(s) pending 10:36:43,943 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader34: running 1 completion(s) 10:36:43,943 AM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - leader34: Persising snapshot at EntryInfo[index=2, term=3]/EntryInfo[index=2, term=3] 10:36:43,943 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader34: support() called 10:36:43,944 AM [leader34-%d689805] [DEBUG] RaftStorage - leader34: starting snapshot writeout to /tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43B-384AEBFF.tmp 10:36:43,944 AM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - leader34: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 10:36:43,944 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader34 (Leader): sendInstallSnapshot 10:36:43,945 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] LeaderInstallSnapshotState - leader34 (Leader): Snapshot 316 bytes, total chunks to send: 1 10:36:43,945 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] LeaderInstallSnapshotState - leader34 (Leader): Next chunk: total length=316, offset=0, size=316, hashCode=-635913009 10:36:43,945 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader34 (Leader): next snapshot chunk size for follower follower35: 316 10:36:43,945 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader34 (Leader): InstallSnapshot sent to follower Vector(user, follower35), Chunk: 1/1 10:36:43,945 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): handleInstallSnapshot: InstallSnapshot{term=3, leaderId=leader34, lastIncludedIndex=2, lastIncludedTerm=3, datasize=316, chunk=1/1, lastChunkHashCode=OptionalInt[-1], votingConfig=VotingConfig{serverInfo=[ServerInfo[peerId=follower35, isVoting=false], ServerInfo[peerId=leader34, isVoting=true]]}} 10:36:43,945 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower35: Lagging 1 entries behind leader leader34 and reached 1 (of expected -1) 10:36:43,946 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotTracker - follower35 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 10:36:43,946 AM [test-pekko.actor.default-dispatcher-14] [INFO] Follower - follower35 (Follower): Snapshot received from leader: leader34 10:36:43,946 AM [test-pekko.actor.default-dispatcher-14] [INFO] SnapshotManager - follower35: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=316}} 10:36:43,946 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower35: support() called 10:36:43,946 AM [leader34-%d689805] [DEBUG] RaftStorage - leader34: finished snapshot writeout to /tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43B-384AEBFF.v1 10:36:43,947 AM [leader34-%d689805] [DEBUG] RaftStorage - leader34: selected SNAPSHOT_V1 to handle file /tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43A-2C3DA5A5.v1 10:36:43,947 AM [leader34-%d689805] [DEBUG] RaftStorage - leader34: selected SNAPSHOT_V1 to handle file /tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43B-384AEBFF.v1 10:36:43,947 AM [leader34-%d689805] [DEBUG] RaftStorage - leader34: skipping unrecognized file /tmp/junit-13059131338678159093/leader34/TermInfo.properties 10:36:43,947 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - follower35: Converted InstallSnapshot from leader: leader34 to state 10:36:43,947 AM [leader34-%d689805] [DEBUG] RaftStorage - leader34: skipping unrecognized file /tmp/junit-13059131338678159093/leader34/journal-v1-1.log 10:36:43,947 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - follower35: last applied journal index prior to persisting applied snapshot: 2 10:36:43,947 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower35: support() called 10:36:43,947 AM [leader34-%d689805] [DEBUG] RaftStorage - leader34: skipping unrecognized file /tmp/junit-13059131338678159093/leader34/metadata-v1-1.log 10:36:43,947 AM [leader34-%d689805] [DEBUG] RaftStorage - leader34: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:42.742237605Z, file=/tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43A-2C3DA5A5.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:43,948 AM [leader34-%d689805] [DEBUG] RaftStorage - leader34: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:43.944434175Z, file=/tmp/junit-13059131338678159093/leader34/snapshot-0000000069AFF43B-384AEBFF.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:43,948 AM [follower35-%d780431] [DEBUG] RaftStorage - follower35: starting snapshot writeout to /tmp/junit-13059131338678159093/follower35/snapshot-0000000069AFF43B-38811D21.tmp 10:36:43,948 AM [leader34-%d689805] [DEBUG] RaftStorageCompleter - leader34: 1 completion(s) pending 10:36:43,948 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader34: running 1 completion(s) 10:36:43,948 AM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - leader34: snapshot is durable as of 2026-03-10T10:36:43.944434175Z 10:36:43,948 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader34: Snapshot success, discarding journal entries up to 3 10:36:43,948 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorSnapshotMessageSupport - leader34: SnapshotComplete received 10:36:43,950 AM [follower35-%d780431] [DEBUG] RaftStorage - follower35: finished snapshot writeout to /tmp/junit-13059131338678159093/follower35/snapshot-0000000069AFF43B-38811D21.v1 10:36:43,950 AM [follower35-%d780431] [DEBUG] RaftStorage - follower35: selected SNAPSHOT_V1 to handle file /tmp/junit-13059131338678159093/follower35/snapshot-0000000069AFF43B-38811D21.v1 10:36:43,950 AM [follower35-%d780431] [DEBUG] RaftStorage - follower35: skipping unrecognized file /tmp/junit-13059131338678159093/follower35/TermInfo.properties 10:36:43,950 AM [follower35-%d780431] [DEBUG] RaftStorage - follower35: skipping unrecognized file /tmp/junit-13059131338678159093/follower35/journal-v1-1.log 10:36:43,950 AM [follower35-%d780431] [DEBUG] RaftStorage - follower35: skipping unrecognized file /tmp/junit-13059131338678159093/follower35/metadata-v1-1.log 10:36:43,950 AM [follower35-%d780431] [DEBUG] RaftStorage - follower35: selected SNAPSHOT_V1 to handle file /tmp/junit-13059131338678159093/follower35/snapshot-0000000069AFF43A-2C538D14.v1 10:36:43,951 AM [follower35-%d780431] [DEBUG] RaftStorage - follower35: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:42.743673108Z, file=/tmp/junit-13059131338678159093/follower35/snapshot-0000000069AFF43A-2C538D14.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:43,951 AM [follower35-%d780431] [DEBUG] RaftStorage - follower35: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:43.947985697Z, file=/tmp/junit-13059131338678159093/follower35/snapshot-0000000069AFF43B-38811D21.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:43,951 AM [follower35-%d780431] [DEBUG] RaftStorageCompleter - follower35: 1 completion(s) pending 10:36:43,951 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower35: running 1 completion(s) 10:36:43,951 AM [test-pekko.actor.default-dispatcher-14] [INFO] SnapshotManager - follower35: snapshot is durable as of 2026-03-10T10:36:43.947985697Z 10:36:43,951 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PeerInfos - follower35: Updated server config: isVoting: false, peers: [PeerInfo [id=leader34, address=pekko://test/user/leader34, votingState=VOTING]] 10:36:43,951 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower35: support() called 10:36:43,951 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower35: applySnapshot called 10:36:43,952 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower35 (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=3, followerId=follower35, chunkIndex=1, success=true} 10:36:43,952 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - follower35: Snapshot success, discarding journal entries up to 2 10:36:43,952 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorSnapshotMessageSupport - follower35: SnapshotComplete received 10:36:43,952 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader34 (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=3, followerId=follower35, chunkIndex=1, success=true} 10:36:43,952 AM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractLeader - leader34 (Leader): Snapshot successfully installed on follower follower35 (last chunk 1) - matchIndex set to 2, nextIndex set to 3 10:36:43,952 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - leader34: fakeSnapshot purging log to 1 for term 3 10:36:43,953 AM [main] [INFO] NonVotingFollowerIntegrationTest - testFollowerResyncWithOneMoreLeaderLogEntryAfterNonPersistentLeaderRestart ending 10:36:43,954 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower35-collector37#1655982213] 10:36:43,966 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower35] 10:36:43,966 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EnabledRaftStorage - follower35: terminating thread VirtualThread[#138,follower35-writer-17]/waiting 10:36:43,966 AM [follower35-writer-17] [DEBUG] JournalWriteTask - follower35: received 1 commands after 1.216 s 10:36:43,967 AM [follower35-writer-17] [DEBUG] JournalWriteTask - follower35: completed 1 commands in 56.77 μs 10:36:43,967 AM [follower35-writer-17] [DEBUG] JournalWriteTask - follower35: journal writer stopped 10:36:43,967 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower35: journal closed 10:36:43,967 AM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - follower35: journal closed 10:36:43,967 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower35: stopped executor with 0 remaining tasks 10:36:43,967 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower35: stopped executor with 0 remaining tasks 10:36:43,968 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader34-collector38#-1324056640] 10:36:43,968 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader34] 10:36:43,969 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader34-collector39#-960508025] 10:36:43,969 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader34] 10:36:43,969 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EnabledRaftStorage - leader34: terminating thread VirtualThread[#140,leader34-writer-19]/waiting 10:36:43,969 AM [leader34-writer-19] [DEBUG] JournalWriteTask - leader34: received 1 commands after 631.4 ms 10:36:43,969 AM [leader34-writer-19] [DEBUG] JournalWriteTask - leader34: completed 1 commands in 24.46 μs 10:36:43,970 AM [leader34-writer-19] [DEBUG] JournalWriteTask - leader34: journal writer stopped 10:36:43,970 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader34: journal closed 10:36:43,970 AM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - leader34: journal closed 10:36:43,970 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: stopped executor with 0 remaining tasks 10:36:43,970 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader34: stopped executor with 0 remaining tasks 10:36:43,977 AM [main] [INFO] NonVotingFollowerIntegrationTest - testFollowerResyncWithMoreLeaderLogEntriesAndDownPeerAfterNonPersistentLeaderRestart starting 10:36:43,979 AM [main] [DEBUG] PropertiesTermInfoStore - leader40: Set currentTerm=1, votedFor=leader40 10:36:43,981 AM [main] [DEBUG] PropertiesTermInfoStore - follower41: Set currentTerm=1, votedFor=leader40 10:36:43,981 AM [main] [DEBUG] RaftStorage - leader40: starting snapshot writeout to /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43B-3A7C84D4.tmp 10:36:43,982 AM [main] [DEBUG] RaftStorage - leader40: finished snapshot writeout to /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43B-3A7C84D4.v1 10:36:43,982 AM [main] [DEBUG] RaftStorage - follower41: starting snapshot writeout to /tmp/junit-16129675699192171389/follower41/snapshot-0000000069AFF43B-3A8FCC7D.tmp 10:36:43,985 AM [main] [DEBUG] RaftStorage - follower41: finished snapshot writeout to /tmp/junit-16129675699192171389/follower41/snapshot-0000000069AFF43B-3A8FCC7D.v1 10:36:43,987 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/follower41] [WARN] [03/10/2026 10:36:43.988] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower41]: org.opendaylight.controller.cluster.common.actor.Monitor@4c97584d 10:36:43,989 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower41: Starting recovery with journal batch size 1000 10:36:43,989 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower41: started executor 10:36:43,989 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower41: started executor 10:36:43,990 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower41: metadata journal open: replayFrom=1 applyTo=0 10:36:43,991 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower41: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:43,991 AM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - follower41: journal open: applyTo=0 10:36:43,991 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower41: recovering to non-persistent 10:36:43,991 AM [follower41-writer-20] [DEBUG] JournalWriteTask - follower41: journal writer started 10:36:43,991 AM [follower41-writer-20] [DEBUG] JournalWriteTask - follower41: waiting to receive commands 10:36:43,992 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower41: Set currentTerm=1, votedFor=leader40 10:36:43,992 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower41: skipping unrecognized file /tmp/junit-16129675699192171389/follower41/TermInfo.properties 10:36:43,992 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower41: skipping unrecognized file /tmp/junit-16129675699192171389/follower41/journal-v1-1.log 10:36:43,992 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower41: skipping unrecognized file /tmp/junit-16129675699192171389/follower41/metadata-v1-1.log 10:36:43,992 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower41: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/follower41/snapshot-0000000069AFF43B-3A8FCC7D.v1 10:36:43,992 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower41: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:43.982502525Z, file=/tmp/junit-16129675699192171389/follower41/snapshot-0000000069AFF43B-3A8FCC7D.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:43,992 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Recovery - follower41: initializing from snapshot taken at 2026-03-10T10:36:43.982502525Z 10:36:43,992 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - follower41: Updated server config: isVoting: false, peers: [PeerInfo [id=leader40, address=pekko://test/user/leader40, votingState=VOTING]] 10:36:43,992 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower41: support() called 10:36:43,993 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower41: Moving last applied index from -1 to -1 10:36:43,993 AM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - follower41: Recovery completed in in 1.245 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:43,993 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower41: Recovery completed and cannot restore from snapshot 10:36:43,995 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/leader40] [WARN] [03/10/2026 10:36:43.996] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader40]: org.opendaylight.controller.cluster.common.actor.Monitor@78d54155 10:36:43,997 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - leader40: Starting recovery with journal batch size 1000 10:36:43,997 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader40: started executor 10:36:43,997 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader40: started executor 10:36:43,998 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader40: metadata journal open: replayFrom=1 applyTo=0 10:36:43,998 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader40: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:43,998 AM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - leader40: journal open: applyTo=0 10:36:43,998 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader40: recovering to non-persistent 10:36:43,998 AM [leader40-writer-21] [DEBUG] JournalWriteTask - leader40: journal writer started 10:36:43,999 AM [leader40-writer-21] [DEBUG] JournalWriteTask - leader40: waiting to receive commands 10:36:43,999 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader40: Set currentTerm=1, votedFor=leader40 10:36:43,999 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader40: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43B-3A7C84D4.v1 10:36:43,999 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/TermInfo.properties 10:36:43,999 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/journal-v1-1.log 10:36:43,999 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/metadata-v1-1.log 10:36:43,999 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader40: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:43.981238996Z, file=/tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43B-3A7C84D4.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:43,999 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Recovery - leader40: initializing from snapshot taken at 2026-03-10T10:36:43.981238996Z 10:36:43,999 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - leader40: Updated server config: isVoting: true, peers: [PeerInfo [id=follower41, address=pekko://test/user/follower41, votingState=NON_VOTING]] 10:36:43,999 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader40: support() called 10:36:43,999 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader40: Moving last applied index from -1 to -1 10:36:44,0 AM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - leader40: Recovery completed in in 1.023 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:44,0 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader40: Recovery completed and cannot restore from snapshot 10:36:44,505 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader40 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.75:17355, Removed) 10:36:44,505 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader40 (Follower): Received ElectionTimeout - switching to Candidate 10:36:44,505 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader40 (Candidate): Election: Candidate has following voting peers: [] 10:36:44,508 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader40: Set currentTerm=2, votedFor=leader40 10:36:44,509 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader40 (Candidate): Starting new election term 2 10:36:44,509 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActorBehavior - leader40 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 10:36:44,509 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader40 (Candidate): Received ElectionTimeout 10:36:44,509 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Election: Leader has following peers: [follower41] 10:36:44,510 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:44,510 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActorBehavior - leader40 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 10:36:44,510 AM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower41 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader40, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 10:36:44,513 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower41: Set currentTerm=2, votedFor=null 10:36:44,513 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower41: Last sync leader does not match current leader leader40, need to catch up to -1 10:36:44,513 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower41 updated: matchIndex: -1, nextIndex: 0 10:36:44,513 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/10/2026 10:36:44.516] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower41] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower41, initialSyncDone=false] 10:36:44,565 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader40: Persist data index=0 term=2 command=zero 10:36:44,565 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader40: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 10:36:44,565 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader40: 1 completion(s) pending 10:36:44,565 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 10:36:44,565 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader40 (Leader): Setting last applied to 0 10:36:44,565 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader40: Moving last applied index from -1 to 0 10:36:44,565 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader40: Applying state for log index 0 data zero 10:36:44,565 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader40: applyState called: zero 10:36:44,566 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:44,566 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 0 is present for follower follower41 10:36:44,566 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Sending AppendEntries to follower follower41: AppendEntries{term=2, leaderId=leader40, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 10:36:44,566 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower41 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader40, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower41 (Follower): Number of entries to be appended = 1 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower41 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower41 (Follower): Append entry to log zero 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower41: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower41: 1 completion(s) pending 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower41 (Follower): Log size is now 1 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower41 (Follower): Commit index set to 0 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower41 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower41, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower41: Lagging 0 entries behind leader leader40 and reached 0 (of expected -1) 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower41 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower41 (Follower): Setting last applied to 0 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower41: Moving last applied index from -1 to 0 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower41: Applying state for log index 0 data zero 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower41: applyState called: zero 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower41: running 1 completion(s) 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower41: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower41: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=2, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@19ed36d} 10:36:44,567 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader40: running 1 completion(s) 10:36:44,567 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower41: synchronized callbacks after 121.4 μs 10:36:44,567 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader40: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} [WARN] [03/10/2026 10:36:44.568] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower41] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower41, initialSyncDone=true] 10:36:44,568 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower41 updated: matchIndex: 0, nextIndex: 1 10:36:44,568 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader40: Persist data index=1 term=2 command=one 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader40: Append log entry and persist JournaledLogEntry{index=1, term=2, command=one, pending=true} 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader40: 1 completion(s) pending 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Replicate message: identifier: MockIdentifier{value=one}, logIndex: 1, isSendImmediate: true 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader40 (Leader): Setting last applied to 1 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader40: Moving last applied index from 0 to 1 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader40: Applying state for log index 1 data one 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader40: applyState called: one 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 1 is present for follower follower41 10:36:44,569 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Sending AppendEntries to follower follower41: AppendEntries{term=2, leaderId=leader40, prevLogIndex=0, prevLogTerm=2, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=one, pending=true}]} 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower41 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader40, prevLogIndex=0, prevLogTerm=2, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=one, pending=true}]} 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower41 (Follower): Number of entries to be appended = 1 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower41 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower41 (Follower): Append entry to log one 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower41: Append log entry and persist JournaledLogEntry{index=1, term=2, command=one, pending=true} 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower41: 1 completion(s) pending 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower41 (Follower): Log size is now 2 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower41 (Follower): Commit index set to 1 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower41 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower41, logLastIndex=1, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower41: Lagging 0 entries behind leader leader40 and reached 1 (of expected -1) 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower41 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower41 (Follower): Setting last applied to 1 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower41: Moving last applied index from 0 to 1 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower41: Applying state for log index 1 data one 10:36:44,570 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower41: applyState called: one 10:36:44,571 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower41: running 1 completion(s) 10:36:44,571 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower41: persist complete JournaledLogEntry{index=1, term=2, command=one, pending=false} 10:36:44,571 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower41: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=2, command=one, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@4496b264} 10:36:44,571 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader40: running 1 completion(s) 10:36:44,571 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader40: persist complete JournaledLogEntry{index=1, term=2, command=one, pending=true} 10:36:44,571 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - leader40: fakeSnapshot purging log to 0 for term 2 10:36:44,571 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower41: synchronized callbacks after 150.0 μs 10:36:44,571 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower41 updated: matchIndex: 1, nextIndex: 2 10:36:44,571 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 10:36:44,571 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader40: Persist data index=2 term=2 command=two 10:36:44,572 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader40: Append log entry and persist JournaledLogEntry{index=2, term=2, command=two, pending=true} 10:36:44,572 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader40: 1 completion(s) pending 10:36:44,572 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Replicate message: identifier: MockIdentifier{value=two}, logIndex: 2, isSendImmediate: true 10:36:44,572 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader40 (Leader): Setting last applied to 2 10:36:44,572 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader40: Moving last applied index from 1 to 2 10:36:44,572 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader40: Applying state for log index 2 data two 10:36:44,572 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader40: applyState called: two 10:36:44,572 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 10:36:44,572 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 2 is present for follower follower41 10:36:44,572 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Sending AppendEntries to follower follower41: AppendEntries{term=2, leaderId=leader40, prevLogIndex=1, prevLogTerm=2, leaderCommit=2, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=2, command=two, pending=true}]} 10:36:44,573 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader40: running 1 completion(s) 10:36:44,573 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader40: persist complete JournaledLogEntry{index=2, term=2, command=two, pending=true} 10:36:44,573 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - leader40: fakeSnapshot purging log to 1 for term 2 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader40, prevLogIndex=1, prevLogTerm=2, leaderCommit=2, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=2, command=two, pending=false}]} 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): Number of entries to be appended = 1 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): Append entry to log two 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower41: Append log entry and persist JournaledLogEntry{index=2, term=2, command=two, pending=false} 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower41: 1 completion(s) pending 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): Log size is now 3 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): Commit index set to 2 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower41, logLastIndex=2, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower41: Lagging 0 entries behind leader leader40 and reached 2 (of expected -1) 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 2 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower41 (Follower): Setting last applied to 2 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower41: Moving last applied index from 1 to 2 10:36:44,573 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower41: Applying state for log index 2 data two 10:36:44,579 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower41: applyState called: two 10:36:44,580 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower41 updated: matchIndex: 2, nextIndex: 3 10:36:44,580 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 1 10:36:44,580 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower41: fakeSnapshot purging log to 0 for term 2 10:36:44,580 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower41: running 1 completion(s) 10:36:44,580 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower41: persist complete JournaledLogEntry{index=2, term=2, command=two, pending=false} 10:36:44,580 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower41: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=2, command=two, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@634c6b7b} 10:36:44,580 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - follower41: synchronized callbacks after 138.8 μs 10:36:44,745 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower41: Lagging 0 entries behind leader leader40 and reached 2 (of expected -1) 10:36:44,745 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - follower41: fakeSnapshot purging log to 1 for term 2 10:36:44,768 AM [main] [DEBUG] RaftStorage - leader40: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43B-3A7C84D4.v1 10:36:44,769 AM [main] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/TermInfo.properties 10:36:44,769 AM [main] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/journal-v1-1.log 10:36:44,769 AM [main] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/metadata-v1-1.log 10:36:44,769 AM [main] [DEBUG] RaftStorage - leader40: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:43.981238996Z, file=/tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43B-3A7C84D4.v1, size=88, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:44,781 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EnabledRaftStorage - leader40: terminating thread VirtualThread[#145,leader40-writer-21]/waiting 10:36:44,781 AM [leader40-writer-21] [DEBUG] JournalWriteTask - leader40: received 1 commands after 782.7 ms 10:36:44,781 AM [leader40-writer-21] [DEBUG] JournalWriteTask - leader40: completed 1 commands in 32.50 μs 10:36:44,782 AM [leader40-writer-21] [DEBUG] JournalWriteTask - leader40: journal writer stopped 10:36:44,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - leader40: journal closed 10:36:44,782 AM [test-pekko.actor.default-dispatcher-14] [INFO] EnabledRaftStorage - leader40: journal closed 10:36:44,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: stopped executor with 0 remaining tasks 10:36:44,782 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: stopped executor with 0 remaining tasks 10:36:44,783 AM [main] [DEBUG] RaftStorage - leader40: starting snapshot writeout to /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2EAF026A.tmp 10:36:44,785 AM [main] [DEBUG] RaftStorage - leader40: finished snapshot writeout to /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2EAF026A.v1 10:36:44,785 AM [main] [DEBUG] RaftStorage - follower42: starting snapshot writeout to /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2ED0E724.tmp 10:36:44,787 AM [main] [DEBUG] RaftStorage - follower42: finished snapshot writeout to /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2ED0E724.v1 10:36:44,790 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/follower42] [WARN] [03/10/2026 10:36:44.791] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower42]: org.opendaylight.controller.cluster.common.actor.Monitor@50e24b13 10:36:44,792 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActor - follower42: Starting recovery with journal batch size 1000 10:36:44,792 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower42: started executor 10:36:44,793 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower42: started executor 10:36:44,794 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - follower42: metadata journal open: replayFrom=1 applyTo=0 10:36:44,795 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - follower42: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:44,795 AM [test-pekko.actor.default-dispatcher-14] [INFO] EnabledRaftStorage - follower42: journal open: applyTo=0 10:36:44,795 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - follower42: recovering to non-persistent 10:36:44,795 AM [follower42-writer-22] [DEBUG] JournalWriteTask - follower42: journal writer started 10:36:44,795 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - /tmp/junit-16129675699192171389/follower42/TermInfo.properties does not exist 10:36:44,795 AM [follower42-writer-22] [DEBUG] JournalWriteTask - follower42: waiting to receive commands 10:36:44,797 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - follower42: Set currentTerm=0, votedFor=null 10:36:44,797 AM [test-pekko.actor.default-dispatcher-14] [INFO] Recovery - follower42: Local TermInfo store seeded with TermInfo{term=0} 10:36:44,797 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower42: skipping unrecognized file /tmp/junit-16129675699192171389/follower42/TermInfo.properties 10:36:44,797 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower42: skipping unrecognized file /tmp/junit-16129675699192171389/follower42/journal-v1-1.log 10:36:44,797 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower42: skipping unrecognized file /tmp/junit-16129675699192171389/follower42/metadata-v1-1.log 10:36:44,797 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower42: no eligible files found 10:36:44,797 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - follower42: Moving last applied index from -1 to -1 10:36:44,797 AM [test-pekko.actor.default-dispatcher-14] [INFO] Recovery - follower42: Recovery completed in in 2.396 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:44,797 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - follower42: Recovery completed and can restore from snapshot 10:36:44,799 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - Actor created TestActor[pekko://test/user/leader40] [WARN] [03/10/2026 10:36:44.800] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader40]: org.opendaylight.controller.cluster.common.actor.Monitor@29c4f86c 10:36:44,801 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActor - leader40: Starting recovery with journal batch size 1000 10:36:44,801 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: started executor 10:36:44,801 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: started executor 10:36:44,802 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - leader40: metadata journal open: replayFrom=1 applyTo=0 10:36:44,802 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - leader40: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 10:36:44,802 AM [test-pekko.actor.default-dispatcher-14] [INFO] EnabledRaftStorage - leader40: journal open: applyTo=0 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader40: recovering to non-persistent 10:36:44,803 AM [leader40-writer-23] [DEBUG] JournalWriteTask - leader40: journal writer started 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader40: Set currentTerm=2, votedFor=leader40 10:36:44,803 AM [leader40-writer-23] [DEBUG] JournalWriteTask - leader40: waiting to receive commands 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43B-3A7C84D4.v1 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2ED0E724.v1 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/TermInfo.properties 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/journal-v1-1.log 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2EAF026A.v1 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/metadata-v1-1.log 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:44.785442596Z, file=/tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2ED0E724.v1, size=112, entryCompress=NONE, stateCompress=NONE} as the latest file 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Recovery - leader40: initializing from snapshot taken at 2026-03-10T10:36:44.785442596Z 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PeerInfos - leader40: Updated server config: isVoting: true, peers: [PeerInfo [id=follower42, address=pekko://test/user/follower42, votingState=VOTING], PeerInfo [id=downPeer, address=null, votingState=NON_VOTING], PeerInfo [id=follower41, address=pekko://test/user/follower41, votingState=NON_VOTING]] 10:36:44,803 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader40: support() called 10:36:44,804 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader40: Moving last applied index from -1 to -1 10:36:44,804 AM [test-pekko.actor.default-dispatcher-14] [INFO] Recovery - leader40: Recovery completed in in 1.300 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 10:36:44,804 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader40: Recovery completed and cannot restore from snapshot 10:36:45,285 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader40 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.75:17355, Removed) 10:36:45,285 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader40 (Follower): Received ElectionTimeout - switching to Candidate 10:36:45,285 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader40 (Candidate): Election: Candidate has following voting peers: [follower42] 10:36:45,288 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader40: Set currentTerm=3, votedFor=leader40 10:36:45,288 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader40 (Candidate): Starting new election term 3 10:36:45,288 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader40 (Candidate): Sending RequestVote{term=3, candidateId=leader40, lastLogIndex=-1, lastLogTerm=-1} to peer follower42 10:36:45,289 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActorBehavior - leader40 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 10:36:45,289 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower42 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 10:36:45,289 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower42 (Follower): Cluster state: [] 10:36:45,289 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower42 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=leader40, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 10:36:45,289 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower42 (Follower): Term 3 in "RequestVote{term=3, candidateId=leader40, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 0 - updating term 10:36:45,291 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower42: Set currentTerm=3, votedFor=null 10:36:45,292 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower42 (Follower): In requestVote: RequestVote{term=3, candidateId=leader40, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 10:36:45,294 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower42: Set currentTerm=3, votedFor=leader40 10:36:45,295 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower42 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 10:36:45,295 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader40 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 10:36:45,295 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader40 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 10:36:45,295 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Election: Leader has following peers: [follower42, downPeer, follower41] 10:36:45,295 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:45,296 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 10:36:45,296 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - leader40 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 10:36:45,296 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower42: Last sync leader does not match current leader leader40, need to catch up to -1 10:36:45,296 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower42 updated: matchIndex: -1, nextIndex: 0 10:36:45,296 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [03/10/2026 10:36:45.296] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower42] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower42, initialSyncDone=false] 10:36:45,315 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader40: Persist data index=0 term=3 command=zero-1 10:36:45,315 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - leader40: Append log entry and persist JournaledLogEntry{index=0, term=3, command=zero-1, pending=true} 10:36:45,315 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader40: 1 completion(s) pending 10:36:45,315 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Replicate message: identifier: MockIdentifier{value=zero-1}, logIndex: 0, isSendImmediate: true 10:36:45,316 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:45,316 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 0 is present for follower follower42 10:36:45,316 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Sending AppendEntries to follower follower42: AppendEntries{term=3, leaderId=leader40, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=3, command=zero-1, pending=true}]} 10:36:45,316 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:45,316 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=leader40, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=3, command=zero-1, pending=true}]} 10:36:45,316 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): Number of entries to be appended = 1 10:36:45,316 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 10:36:45,316 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader40: running 1 completion(s) 10:36:45,316 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): Append entry to log zero-1 10:36:45,316 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - follower42: Append log entry and persist JournaledLogEntry{index=0, term=3, command=zero-1, pending=true} 10:36:45,316 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - leader40: persist complete JournaledLogEntry{index=0, term=3, command=zero-1, pending=true} 10:36:45,316 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower42: 1 completion(s) pending 10:36:45,317 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): Log size is now 1 10:36:45,317 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower42, logLastIndex=0, logLastTerm=3, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:45,317 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower42: Lagging 0 entries behind leader leader40 and reached -1 (of expected -1) 10:36:45,317 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower42: running 1 completion(s) 10:36:45,317 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - follower42: persist complete JournaledLogEntry{index=0, term=3, command=zero-1, pending=false} 10:36:45,317 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower42: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=3, command=zero-1, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@3066d87e} 10:36:45,317 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower42: synchronized callbacks after 165.6 μs [WARN] [03/10/2026 10:36:45.317] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower42] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower42, initialSyncDone=true] 10:36:45,319 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower42 updated: matchIndex: 0, nextIndex: 1 10:36:45,319 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 10:36:45,319 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader40 (Leader): Setting last applied to 0 10:36:45,319 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader40: Moving last applied index from -1 to 0 10:36:45,319 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader40: Applying state for log index 0 data zero-1 10:36:45,319 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - leader40: applyState called: zero-1 10:36:45,319 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 10:36:45,320 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): Commit index set to 0 10:36:45,320 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader40: Persist data index=1 term=3 command=one-1 10:36:45,320 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower42: Lagging 0 entries behind leader leader40 and reached 0 (of expected -1) 10:36:45,320 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 10:36:45,320 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - leader40: Append log entry and persist JournaledLogEntry{index=1, term=3, command=one-1, pending=true} 10:36:45,320 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - follower42 (Follower): Setting last applied to 0 10:36:45,320 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - follower42: Moving last applied index from -1 to 0 10:36:45,320 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader40: 1 completion(s) pending 10:36:45,320 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - follower42: Applying state for log index 0 data zero-1 10:36:45,320 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower42: applyState called: zero-1 10:36:45,320 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Replicate message: identifier: MockIdentifier{value=one-1}, logIndex: 1, isSendImmediate: true 10:36:45,320 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 10:36:45,320 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 1 is present for follower follower42 10:36:45,320 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Sending AppendEntries to follower follower42: AppendEntries{term=3, leaderId=leader40, prevLogIndex=0, prevLogTerm=3, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=3, command=one-1, pending=true}]} 10:36:45,321 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: false, followerNextIndex: -1, leaderLastIndex: 1, leaderSnapShotIndex: -1 10:36:45,321 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=leader40, prevLogIndex=0, prevLogTerm=3, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=3, command=one-1, pending=true}]} 10:36:45,321 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Number of entries to be appended = 1 10:36:45,321 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 10:36:45,321 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader40: running 1 completion(s) 10:36:45,321 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Append entry to log one-1 10:36:45,321 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader40: persist complete JournaledLogEntry{index=1, term=3, command=one-1, pending=true} 10:36:45,321 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower42: Append log entry and persist JournaledLogEntry{index=1, term=3, command=one-1, pending=true} 10:36:45,321 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: 1 completion(s) pending 10:36:45,321 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Log size is now 2 10:36:45,321 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower42, logLastIndex=1, logLastTerm=3, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:45,321 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower42: Lagging 0 entries behind leader leader40 and reached 0 (of expected -1) 10:36:45,321 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: running 1 completion(s) 10:36:45,321 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 1 is present for follower follower42 10:36:45,322 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower42: persist complete JournaledLogEntry{index=1, term=3, command=one-1, pending=false} 10:36:45,322 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=3, command=one-1, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@4ec43d7b} 10:36:45,322 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower42 updated: matchIndex: 1, nextIndex: 2 10:36:45,322 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 10:36:45,322 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: synchronized callbacks after 324.8 μs 10:36:45,322 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader40 (Leader): Setting last applied to 1 10:36:45,322 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader40: Moving last applied index from 0 to 1 10:36:45,322 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader40: Applying state for log index 1 data one-1 10:36:45,322 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader40: applyState called: one-1 10:36:45,322 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: -1 10:36:45,322 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader40: Persist data index=2 term=3 command=two-1 10:36:45,322 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader40: Append log entry and persist JournaledLogEntry{index=2, term=3, command=two-1, pending=true} 10:36:45,322 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Commit index set to 1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader40: 1 completion(s) pending 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower42: Lagging 0 entries behind leader leader40 and reached 1 (of expected -1) 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Replicate message: identifier: MockIdentifier{value=two-1}, logIndex: 2, isSendImmediate: true 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower42 (Follower): Setting last applied to 1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower42: Moving last applied index from 0 to 1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower42: Applying state for log index 1 data one-1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: -1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower42: applyState called: one-1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 2 is present for follower follower42 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Sending AppendEntries to follower follower42: AppendEntries{term=3, leaderId=leader40, prevLogIndex=1, prevLogTerm=3, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=3, command=two-1, pending=true}]} 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: false, followerNextIndex: -1, leaderLastIndex: 2, leaderSnapShotIndex: -1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader40: running 1 completion(s) 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader40: persist complete JournaledLogEntry{index=2, term=3, command=two-1, pending=true} 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 2 is present for follower follower42 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=leader40, prevLogIndex=1, prevLogTerm=3, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=3, command=two-1, pending=false}]} 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Number of entries to be appended = 1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader40: Persist data index=3 term=3 command=three-1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader40: Append log entry and persist JournaledLogEntry{index=3, term=3, command=three-1, pending=true} 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Append entry to log two-1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader40: 1 completion(s) pending 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower42: Append log entry and persist JournaledLogEntry{index=2, term=3, command=two-1, pending=false} 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Replicate message: identifier: MockIdentifier{value=three-1}, logIndex: 3, isSendImmediate: true 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: 1 completion(s) pending 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 2, leaderLastIndex: 3, leaderSnapShotIndex: -1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Log size is now 3 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 2 is present for follower follower42 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower42, logLastIndex=2, logLastTerm=3, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:45,323 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: false, followerNextIndex: -1, leaderLastIndex: 3, leaderSnapShotIndex: -1 10:36:45,323 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower42: Lagging 0 entries behind leader leader40 and reached 1 (of expected -1) 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader40: running 1 completion(s) 10:36:45,324 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: running 1 completion(s) 10:36:45,324 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower42: persist complete JournaledLogEntry{index=2, term=3, command=two-1, pending=false} 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - leader40: persist complete JournaledLogEntry{index=3, term=3, command=three-1, pending=true} 10:36:45,324 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=3, command=two-1, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@3fd0fe98} 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower42 updated: matchIndex: 2, nextIndex: 3 10:36:45,324 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: synchronized callbacks after 218.1 μs 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - leader40 (Leader): Setting last applied to 2 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader40: Moving last applied index from 1 to 2 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader40: Applying state for log index 2 data two-1 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader40: applyState called: two-1 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 3, leaderLastIndex: 3, leaderSnapShotIndex: -1 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 3 is present for follower follower42 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Sending AppendEntries to follower follower42: AppendEntries{term=3, leaderId=leader40, prevLogIndex=2, prevLogTerm=3, leaderCommit=2, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=3, command=three-1, pending=false}]} 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=leader40, prevLogIndex=2, prevLogTerm=3, leaderCommit=2, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=3, command=three-1, pending=false}]} 10:36:45,324 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader40: Persist data index=4 term=3 command=four-1 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): Number of entries to be appended = 1 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 10:36:45,324 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader40: Append log entry and persist JournaledLogEntry{index=4, term=3, command=four-1, pending=true} 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): Append entry to log three-1 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - follower42: Append log entry and persist JournaledLogEntry{index=3, term=3, command=three-1, pending=false} 10:36:45,324 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader40: 1 completion(s) pending 10:36:45,324 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower42: 1 completion(s) pending 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): Log size is now 4 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Replicate message: identifier: MockIdentifier{value=four-1}, logIndex: 4, isSendImmediate: true 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): Commit index set to 2 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower42, logLastIndex=3, logLastTerm=3, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower42: Lagging 0 entries behind leader leader40 and reached 2 (of expected -1) 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower42 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 3 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorBehavior - follower42 (Follower): Setting last applied to 2 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - follower42: Moving last applied index from 1 to 2 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - follower42: Applying state for log index 2 data two-1 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower42: applyState called: two-1 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 3, leaderLastIndex: 4, leaderSnapShotIndex: -1 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 3 is present for follower follower42 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: false, followerNextIndex: -1, leaderLastIndex: 4, leaderSnapShotIndex: -1 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower42: running 1 completion(s) 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] ReplicatedLogImpl - follower42: persist complete JournaledLogEntry{index=3, term=3, command=three-1, pending=false} 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower42: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=3, term=3, command=three-1, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@5d0644d7} 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader40: running 1 completion(s) 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader40: persist complete JournaledLogEntry{index=4, term=3, command=four-1, pending=true} 10:36:45,325 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower42: synchronized callbacks after 89.80 μs 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower42 updated: matchIndex: 3, nextIndex: 4 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader40 (Leader): Setting last applied to 3 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader40: Moving last applied index from 2 to 3 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader40: Applying state for log index 3 data three-1 10:36:45,325 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader40: applyState called: three-1 10:36:45,326 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 4, leaderLastIndex: 4, leaderSnapShotIndex: -1 10:36:45,326 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 4 is present for follower follower42 10:36:45,326 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader40 (Leader): Sending AppendEntries to follower follower42: AppendEntries{term=3, leaderId=leader40, prevLogIndex=3, prevLogTerm=3, leaderCommit=3, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=3, command=four-1, pending=false}]} 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=leader40, prevLogIndex=3, prevLogTerm=3, leaderCommit=3, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=4, term=3, command=four-1, pending=false}]} 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Number of entries to be appended = 1 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): After cleanup, lastIndex: 3, entries to be added from: 0 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Append entry to log four-1 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower42: Append log entry and persist JournaledLogEntry{index=4, term=3, command=four-1, pending=false} 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: 1 completion(s) pending 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Log size is now 5 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): Commit index set to 3 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower42, logLastIndex=4, logLastTerm=3, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower42: Lagging 0 entries behind leader leader40 and reached 3 (of expected -1) 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower42 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): 2, lastIndex(): 4 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower42 (Follower): Setting last applied to 3 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower42: Moving last applied index from 2 to 3 10:36:45,326 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower42 updated: matchIndex: 4, nextIndex: 5 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower42: Applying state for log index 3 data three-1 10:36:45,326 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower42: applyState called: three-1 10:36:45,326 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader40 (Leader): Applying to log - commitIndex: 4, lastAppliedIndex: 3 10:36:45,327 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader40 (Leader): Setting last applied to 4 10:36:45,327 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader40: Moving last applied index from 3 to 4 10:36:45,327 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader40: Applying state for log index 4 data four-1 10:36:45,327 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader40: applyState called: four-1 10:36:45,327 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: running 1 completion(s) 10:36:45,327 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower42: persist complete JournaledLogEntry{index=4, term=3, command=four-1, pending=false} 10:36:45,327 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=4, term=3, command=four-1, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c08d7150@1207f2f2} 10:36:45,327 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower42: synchronized callbacks after 125.9 μs 10:36:45,327 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower42: active: true, followerNextIndex: 5, leaderLastIndex: 4, leaderSnapShotIndex: -1 10:36:45,327 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower42 (Follower): Commit index set to 4 10:36:45,327 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower42: Lagging 0 entries behind leader leader40 and reached 4 (of expected -1) 10:36:45,327 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower42 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 4, context.getLastApplied(): 3, lastIndex(): 4 10:36:45,328 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower42 (Follower): Setting last applied to 4 10:36:45,328 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower42: Moving last applied index from 3 to 4 10:36:45,328 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower42: Applying state for log index 4 data four-1 10:36:45,328 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower42: applyState called: four-1 10:36:45,415 AM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower41 (Follower): Term 3 in "AppendEntries{term=3, leaderId=leader40, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 10:36:45,420 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower41: Set currentTerm=3, votedFor=null 10:36:45,420 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower41: Lagging -3 entries behind leader leader40 and reached 2 (of expected -1) 10:36:45,421 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractLeader - leader40 (Leader): handleAppendEntriesReply: follower follower41 last log term 2 for index 2 conflicts with the leader's 3 - set the follower's next index to 1 10:36:45,421 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: true, followerNextIndex: 1, leaderLastIndex: 4, leaderSnapShotIndex: -1 10:36:45,421 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 1 is present for follower follower41 10:36:45,421 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Sending AppendEntries to follower follower41: AppendEntries{term=3, leaderId=leader40, prevLogIndex=0, prevLogTerm=3, leaderCommit=4, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=3, command=one-1, pending=false}, JournaledLogEntry{index=2, term=3, command=two-1, pending=false}, JournaledLogEntry{index=3, term=3, command=three-1, pending=false}, JournaledLogEntry{index=4, term=3, command=four-1, pending=false}]} 10:36:45,421 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower41 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=leader40, prevLogIndex=0, prevLogTerm=3, leaderCommit=4, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=3, command=one-1, pending=false}, JournaledLogEntry{index=2, term=3, command=two-1, pending=false}, JournaledLogEntry{index=3, term=3, command=three-1, pending=false}, JournaledLogEntry{index=4, term=3, command=four-1, pending=false}]} 10:36:45,421 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower41 (Follower): The prevLogIndex 0 was found in the log but the term 2 is not equal to the append entries prevLogTerm 3 - lastIndex: 2, snapshotIndex: 1, snapshotTerm: 2 10:36:45,422 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower41 (Follower): Follower is out-of-sync so sending negative reply: AppendEntriesReply{term=3, success=false, followerId=follower41, logLastIndex=2, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 10:36:45,422 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower41: Lagging 2 entries behind leader leader40 and reached 2 (of expected -1) 10:36:45,422 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - received unsuccessful reply: AppendEntriesReply{term=3, success=false, followerId=follower41, logLastIndex=2, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, leader snapshotIndex: -1, snapshotTerm: -1, replicatedToAllIndex: -1 10:36:45,422 AM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractLeader - leader40 (Leader): follower follower41 last log term 2 conflicts with the leader's 3 - dec next index to 0 10:36:45,422 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Checking sendAppendEntries for follower follower41: active: true, followerNextIndex: 0, leaderLastIndex: 4, leaderSnapShotIndex: -1 10:36:45,422 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): sendAppendEntries: 0 is present for follower follower41 10:36:45,422 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader40 (Leader): Sending AppendEntries to follower follower41: AppendEntries{term=3, leaderId=leader40, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=4, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=3, command=zero-1, pending=false}, JournaledLogEntry{index=1, term=3, command=one-1, pending=false}, JournaledLogEntry{index=2, term=3, command=two-1, pending=false}, JournaledLogEntry{index=3, term=3, command=three-1, pending=false}, JournaledLogEntry{index=4, term=3, command=four-1, pending=false}]} 10:36:45,423 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=leader40, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=4, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=3, command=zero-1, pending=false}, JournaledLogEntry{index=1, term=3, command=one-1, pending=false}, JournaledLogEntry{index=2, term=3, command=two-1, pending=false}, JournaledLogEntry{index=3, term=3, command=three-1, pending=false}, JournaledLogEntry{index=4, term=3, command=four-1, pending=false}]} 10:36:45,423 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): Number of entries to be appended = 5 10:36:45,423 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): matchEntry JournaledLogEntry{index=0, term=3, command=zero-1, pending=false} is present: existingEntryTerm: -1 10:36:45,423 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower41 (Follower): matchEntry JournaledLogEntry{index=1, term=3, command=one-1, pending=false} is present: existingEntryTerm: 2 10:36:45,423 AM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower41 (Follower): Removing entries from log starting at 1, commitIndex: 2, lastApplied: 2 10:36:45,423 AM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower41 (Follower): Could not remove entries - sending reply to force snapshot 10:36:45,423 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower41: Lagging 2 entries behind leader leader40 and reached 2 (of expected -1) 10:36:45,423 AM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractLeader - leader40 (Leader): handleAppendEntriesReply - received unsuccessful reply: AppendEntriesReply{term=3, success=false, followerId=follower41, logLastIndex=2, logLastTerm=2, forceInstallSnapshot=true, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5}, leader snapshotIndex: -1, snapshotTerm: -1, replicatedToAllIndex: -1 10:36:45,423 AM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader40: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=4, lastAppliedTerm=3, lastIndex=4, lastTerm=3, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower41 10:36:45,423 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader40: last applied journal index prior to capture: 5 10:36:45,423 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader40: support() called 10:36:45,424 AM [leader40-%d185764] [DEBUG] RaftStorageCompleter - leader40: 1 completion(s) pending 10:36:45,425 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader40: running 1 completion(s) 10:36:45,425 AM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader40: Persising snapshot at EntryInfo[index=4, term=3]/EntryInfo[index=4, term=3] 10:36:45,425 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader40: support() called 10:36:45,425 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: starting snapshot writeout to /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43D-195D16F8.tmp 10:36:45,426 AM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader40: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 10:36:45,426 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader40 (Leader): sendInstallSnapshot 10:36:45,426 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] LeaderInstallSnapshotState - leader40 (Leader): Snapshot 355 bytes, total chunks to send: 1 10:36:45,426 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] LeaderInstallSnapshotState - leader40 (Leader): Next chunk: total length=355, offset=0, size=355, hashCode=951985329 10:36:45,426 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader40 (Leader): next snapshot chunk size for follower follower41: 355 10:36:45,426 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader40 (Leader): InstallSnapshot sent to follower Vector(user, follower41), Chunk: 1/1 10:36:45,426 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower41 (Follower): handleInstallSnapshot: InstallSnapshot{term=3, leaderId=leader40, lastIncludedIndex=4, lastIncludedTerm=3, datasize=355, chunk=1/1, lastChunkHashCode=OptionalInt[-1], votingConfig=VotingConfig{serverInfo=[ServerInfo[peerId=follower42, isVoting=true], ServerInfo[peerId=downPeer, isVoting=false], ServerInfo[peerId=follower41, isVoting=false], ServerInfo[peerId=leader40, isVoting=true]]}} 10:36:45,426 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower41: Lagging 2 entries behind leader leader40 and reached 2 (of expected -1) 10:36:45,426 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotTracker - follower41 (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 10:36:45,426 AM [test-pekko.actor.default-dispatcher-14] [INFO] Follower - follower41 (Follower): Snapshot received from leader: leader40 10:36:45,426 AM [test-pekko.actor.default-dispatcher-14] [INFO] SnapshotManager - follower41: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=355}} 10:36:45,426 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower41: support() called 10:36:45,427 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: finished snapshot writeout to /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43D-195D16F8.v1 10:36:45,427 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - follower41: Converted InstallSnapshot from leader: leader40 to state 10:36:45,427 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43B-3A7C84D4.v1 10:36:45,427 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - follower41: last applied journal index prior to persisting applied snapshot: 3 10:36:45,427 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower41: support() called 10:36:45,427 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2ED0E724.v1 10:36:45,428 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43D-195D16F8.v1 10:36:45,428 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/TermInfo.properties 10:36:45,428 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/journal-v1-1.log 10:36:45,428 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2EAF026A.v1 10:36:45,428 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: skipping unrecognized file /tmp/junit-16129675699192171389/leader40/metadata-v1-1.log 10:36:45,428 AM [follower41-%d416505] [DEBUG] RaftStorage - follower41: starting snapshot writeout to /tmp/junit-16129675699192171389/follower41/snapshot-0000000069AFF43D-19857B3F.tmp 10:36:45,428 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:43.981238996Z, file=/tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43B-3A7C84D4.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:45,428 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:44.783221354Z, file=/tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2EAF026A.v1, size=112, entryCompress=NONE, stateCompress=NONE} 10:36:45,428 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:44.785442596Z, file=/tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43C-2ED0E724.v1, size=112, entryCompress=NONE, stateCompress=NONE} 10:36:45,428 AM [leader40-%d185765] [DEBUG] RaftStorage - leader40: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:45.425531128Z, file=/tmp/junit-16129675699192171389/leader40/snapshot-0000000069AFF43D-195D16F8.v1, size=112, entryCompress=NONE, stateCompress=NONE} 10:36:45,429 AM [leader40-%d185765] [DEBUG] RaftStorageCompleter - leader40: 1 completion(s) pending 10:36:45,429 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - leader40: running 1 completion(s) 10:36:45,429 AM [test-pekko.actor.default-dispatcher-14] [INFO] SnapshotManager - leader40: snapshot is durable as of 2026-03-10T10:36:45.425531128Z 10:36:45,429 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - leader40: Snapshot success, discarding journal entries up to 5 10:36:45,429 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorSnapshotMessageSupport - leader40: SnapshotComplete received 10:36:45,430 AM [follower41-%d416505] [DEBUG] RaftStorage - follower41: finished snapshot writeout to /tmp/junit-16129675699192171389/follower41/snapshot-0000000069AFF43D-19857B3F.v1 10:36:45,431 AM [follower41-%d416505] [DEBUG] RaftStorage - follower41: skipping unrecognized file /tmp/junit-16129675699192171389/follower41/TermInfo.properties 10:36:45,431 AM [follower41-%d416505] [DEBUG] RaftStorage - follower41: skipping unrecognized file /tmp/junit-16129675699192171389/follower41/journal-v1-1.log 10:36:45,431 AM [follower41-%d416505] [DEBUG] RaftStorage - follower41: skipping unrecognized file /tmp/junit-16129675699192171389/follower41/metadata-v1-1.log 10:36:45,431 AM [follower41-%d416505] [DEBUG] RaftStorage - follower41: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/follower41/snapshot-0000000069AFF43B-3A8FCC7D.v1 10:36:45,431 AM [follower41-%d416505] [DEBUG] RaftStorage - follower41: selected SNAPSHOT_V1 to handle file /tmp/junit-16129675699192171389/follower41/snapshot-0000000069AFF43D-19857B3F.v1 10:36:45,431 AM [follower41-%d416505] [DEBUG] RaftStorage - follower41: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:43.982502525Z, file=/tmp/junit-16129675699192171389/follower41/snapshot-0000000069AFF43B-3A8FCC7D.v1, size=88, entryCompress=NONE, stateCompress=NONE} 10:36:45,431 AM [follower41-%d416505] [DEBUG] RaftStorage - follower41: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2026-03-10T10:36:45.428178239Z, file=/tmp/junit-16129675699192171389/follower41/snapshot-0000000069AFF43D-19857B3F.v1, size=112, entryCompress=NONE, stateCompress=NONE} 10:36:45,431 AM [follower41-%d416505] [DEBUG] RaftStorageCompleter - follower41: 1 completion(s) pending 10:36:45,431 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorageCompleter - follower41: running 1 completion(s) 10:36:45,431 AM [test-pekko.actor.default-dispatcher-14] [INFO] SnapshotManager - follower41: snapshot is durable as of 2026-03-10T10:36:45.428178239Z 10:36:45,431 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PeerInfos - follower41: Updated server config: isVoting: false, peers: [PeerInfo [id=follower42, address=null, votingState=VOTING], PeerInfo [id=downPeer, address=null, votingState=NON_VOTING], PeerInfo [id=leader40, address=pekko://test/user/leader40, votingState=VOTING]] 10:36:45,431 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower41: support() called 10:36:45,432 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - follower41: applySnapshot called 10:36:45,432 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - follower41 (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=3, followerId=follower41, chunkIndex=1, success=true} 10:36:45,432 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SnapshotManager - follower41: Snapshot success, discarding journal entries up to 3 10:36:45,432 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorSnapshotMessageSupport - follower41: SnapshotComplete received 10:36:45,432 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader40 (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=3, followerId=follower41, chunkIndex=1, success=true} 10:36:45,432 AM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractLeader - leader40 (Leader): Snapshot successfully installed on follower follower41 (last chunk 1) - matchIndex set to 4, nextIndex set to 5 10:36:45,467 AM [main] [INFO] NonVotingFollowerIntegrationTest - testFollowerResyncWithMoreLeaderLogEntriesAndDownPeerAfterNonPersistentLeaderRestart ending 10:36:45,468 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower41-collector43#-832806289] 10:36:45,480 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower41] 10:36:45,480 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EnabledRaftStorage - follower41: terminating thread VirtualThread[#144,follower41-writer-20]/waiting 10:36:45,481 AM [follower41-writer-20] [DEBUG] JournalWriteTask - follower41: received 1 commands after 1.489 s 10:36:45,481 AM [follower41-writer-20] [DEBUG] JournalWriteTask - follower41: completed 1 commands in 36.91 μs 10:36:45,481 AM [follower41-writer-20] [DEBUG] JournalWriteTask - follower41: journal writer stopped 10:36:45,482 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - follower41: journal closed 10:36:45,482 AM [test-pekko.actor.default-dispatcher-14] [INFO] EnabledRaftStorage - follower41: journal closed 10:36:45,482 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower41: stopped executor with 0 remaining tasks 10:36:45,482 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower41: stopped executor with 0 remaining tasks 10:36:45,482 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader40-collector44#1589497200] 10:36:45,483 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader40] 10:36:45,483 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower42-collector45#1957470100] 10:36:45,483 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower42] 10:36:45,483 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EnabledRaftStorage - follower42: terminating thread VirtualThread[#146,follower42-writer-22]/waiting 10:36:45,484 AM [follower42-writer-22] [DEBUG] JournalWriteTask - follower42: received 1 commands after 688.7 ms 10:36:45,484 AM [follower42-writer-22] [DEBUG] JournalWriteTask - follower42: completed 1 commands in 29.84 μs 10:36:45,484 AM [follower42-writer-22] [DEBUG] JournalWriteTask - follower42: journal writer stopped 10:36:45,484 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - follower42: journal closed 10:36:45,484 AM [test-pekko.actor.default-dispatcher-14] [INFO] EnabledRaftStorage - follower42: journal closed 10:36:45,484 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower42: stopped executor with 0 remaining tasks 10:36:45,484 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower42: stopped executor with 0 remaining tasks 10:36:45,485 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader40-collector46#-868955430] 10:36:45,485 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader40] 10:36:45,485 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EnabledRaftStorage - leader40: terminating thread VirtualThread[#147,leader40-writer-23]/waiting 10:36:45,486 AM [leader40-writer-23] [DEBUG] JournalWriteTask - leader40: received 1 commands after 682.9 ms 10:36:45,486 AM [leader40-writer-23] [DEBUG] JournalWriteTask - leader40: completed 1 commands in 12.93 μs 10:36:45,486 AM [leader40-writer-23] [DEBUG] JournalWriteTask - leader40: journal writer stopped 10:36:45,487 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - leader40: journal closed 10:36:45,487 AM [test-pekko.actor.default-dispatcher-14] [INFO] EnabledRaftStorage - leader40: journal closed 10:36:45,487 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: stopped executor with 0 remaining tasks 10:36:45,487 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader40: stopped executor with 0 remaining tasks 10:36:45,491 AM [test-pekko.actor.default-dispatcher-11] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [INFO] [03/10/2026 10:36:45.490] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 10:36:45,493 AM [test-pekko.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.75:17355] - Exiting completed 10:36:45,493 AM [test-pekko.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.75:17355] - Shutting down... 10:36:45,493 AM [test-pekko.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.75:17355] - Successfully shut down [WARN] [03/10/2026 10:36:45.494] [test-pekko.actor.default-dispatcher-14] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.171.75:17355) [INFO] [03/10/2026 10:36:45.492] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.75:17355] - Exiting completed [INFO] [03/10/2026 10:36:45.492] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.75:17355] - Shutting down... [INFO] [03/10/2026 10:36:45.492] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.75:17355] - Successfully shut down 10:36:45,495 AM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 10:36:45,495 AM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [03/10/2026 10:36:45.495] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.171.75:17355/system/remoting-terminator] Shutting down remote daemon. [INFO] [03/10/2026 10:36:45.495] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.171.75:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 10:36:45,497 AM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [INFO] [03/10/2026 10:36:45.497] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.171.75:17355/system/remoting-terminator] Remoting shut down. [DEBUG] [03/10/2026 10:36:45.498] [test-pekko.actor.internal-dispatcher-3] [EventStream] shutting down: StandardOutLogger