[DEBUG] [07/05/2025 09:27:03.435] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 09:27:03,435 AM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [07/05/2025 09:27:03.435] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [07/05/2025 09:27:03.436] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [07/05/2025 09:27:03.446] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.247:17355] with UID [380783424683647354] 09:27:03,446 AM [test-pekko.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.247:17355] with UID [380783424683647354] [INFO] [07/05/2025 09:27:03.447] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Starting up, Pekko version [1.0.3] ... 09:27:03,448 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [07/05/2025 09:27:03.449] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [07/05/2025 09:27:03.449] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Started up successfully [INFO] [07/05/2025 09:27:03.449] [test-pekko.actor.internal-dispatcher-8] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [07/05/2025 09:27:03.449] [test-pekko.actor.internal-dispatcher-8] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 09:27:03,450 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 09:27:03,450 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Started up successfully 09:27:03,450 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 09:27:03,450 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 09:27:03,459 AM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testReplicationsWithLaggingFollowerCaughtUpViaAppendEntries starting: sending 2 new payloads [WARN] [07/05/2025 09:27:03.460] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower752]: org.opendaylight.controller.cluster.common.actor.Monitor@4d9dca48 [INFO] [pekkoDeadLetter][07/05/2025 09:27:03.460] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower752] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:03,460 AM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower752] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:03,462 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - follower752: Starting recovery with journal batch size 1000 09:27:03,462 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower752: started executor 09:27:03,462 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower752: started executor 09:27:03,462 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower752: metadata journal open: replayFrom=1 applyTo=0 09:27:03,462 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower752: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:03,462 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - follower752: journal open: applyTo=0 09:27:03,463 AM [follower752-writer-148] [DEBUG] JournalWriteTask - follower752: journal writer started 09:27:03,463 AM [follower752-writer-148] [DEBUG] JournalWriteTask - follower752: waiting to receive commands 09:27:03,463 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/follower752/TermInfo.properties does not exist 09:27:03,463 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower752: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/follower752/journal-v1-1.log 09:27:03,463 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower752: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/follower752/metadata-v1-1.log 09:27:03,463 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower752: no eligible files found 09:27:03,464 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower752: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/05/2025 09:27:03.465] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower753]: org.opendaylight.controller.cluster.common.actor.Monitor@175d59ca [INFO] [pekkoDeadLetter][07/05/2025 09:27:03.465] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower753] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:03,465 AM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower753] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:03,465 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - follower753: Starting recovery with journal batch size 1000 09:27:03,465 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower753: started executor 09:27:03,465 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower753: started executor 09:27:03,465 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower753: metadata journal open: replayFrom=1 applyTo=0 09:27:03,465 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower753: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:03,465 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - follower753: journal open: applyTo=0 09:27:03,466 AM [follower753-writer-149] [DEBUG] JournalWriteTask - follower753: journal writer started 09:27:03,466 AM [follower753-writer-149] [DEBUG] JournalWriteTask - follower753: waiting to receive commands 09:27:03,466 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower752: Set currentTerm=0, votedFor=null 09:27:03,466 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower752: Local TermInfo store seeded with TermInfo{term=0} 09:27:03,466 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower752: Pekko recovery completed and can restore from snapshot 09:27:03,466 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower752: starting journal recovery 09:27:03,466 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower752: Moving last applied index from -1 to -1 09:27:03,466 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower752: empty journal: appending 0 entries 09:27:03,466 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower752: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:03,466 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower752: journal recovery completedin 117.4 μs with journalIndex=1 09:27:03,467 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/follower753/TermInfo.properties does not exist 09:27:03,467 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower753: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/follower753/journal-v1-1.log 09:27:03,467 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower753: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/follower753/metadata-v1-1.log 09:27:03,467 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower753: no eligible files found 09:27:03,467 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower753: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:27:03,469 AM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader751] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [07/05/2025 09:27:03.469] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader751]: org.opendaylight.controller.cluster.common.actor.Monitor@3535a9a3 [INFO] [pekkoDeadLetter][07/05/2025 09:27:03.469] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader751] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:03,469 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - leader751: Starting recovery with journal batch size 1000 09:27:03,469 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower753: Set currentTerm=0, votedFor=null 09:27:03,469 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower753: Local TermInfo store seeded with TermInfo{term=0} 09:27:03,470 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower753: Pekko recovery completed and can restore from snapshot 09:27:03,470 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader751: started executor 09:27:03,470 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower753: starting journal recovery 09:27:03,470 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower753: Moving last applied index from -1 to -1 09:27:03,470 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower753: empty journal: appending 0 entries 09:27:03,470 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader751: started executor 09:27:03,470 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower753: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:03,470 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower753: journal recovery completedin 113.8 μs with journalIndex=1 09:27:03,470 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader751: metadata journal open: replayFrom=1 applyTo=0 09:27:03,470 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader751: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:03,470 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader751: journal open: applyTo=0 09:27:03,470 AM [leader751-writer-150] [DEBUG] JournalWriteTask - leader751: journal writer started 09:27:03,470 AM [leader751-writer-150] [DEBUG] JournalWriteTask - leader751: waiting to receive commands 09:27:03,471 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/leader751/TermInfo.properties does not exist 09:27:03,471 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader751: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/leader751/journal-v1-1.log 09:27:03,471 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader751: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/leader751/metadata-v1-1.log 09:27:03,471 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader751: no eligible files found 09:27:03,471 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=5, votedFor=leader751 09:27:03,471 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - leader751: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:27:03,473 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=5, votedFor=leader751 09:27:03,473 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - leader751: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader751} 09:27:03,473 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - leader751: taking snapshot to clear Pekko persistence to 1 09:27:03,473 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Recovery - leader751: no applied entries in recovery log, re-snapshotting EntryInfo[index=-1, term=-1] 09:27:03,473 AM [test-pekko.actor.default-dispatcher-14] [INFO] Recovery - leader751: Taking snapshot on entry with index -1 09:27:03,473 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader751: support() called 09:27:03,473 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader751: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/leader751/snapshot-000000006868EFE7-1C3D8E9B.tmp 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader751: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/leader751/snapshot-000000006868EFE7-1C3D8E9B.v1 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader751: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/leader751/TermInfo.properties 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader751: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/leader751/journal-v1-1.log 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader751: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/leader751/metadata-v1-1.log 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader751: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/leader751/snapshot-000000006868EFE7-1C3D8E9B.v1 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader751: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-05T09:27:03.473796251Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14290616152390522042/leader751/snapshot-000000006868EFE7-1C3D8E9B.v1, size=221, entryCompress=NONE, stateCompress=NONE} 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [INFO] Recovery - leader751: Snapshot completed in 4.433 ms, resetting timer for the next recovery snapshot 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader751: Pekko recovery completed and can restore from snapshot 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader751: starting journal recovery 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader751: Moving last applied index from -1 to -1 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] JournalRecovery - leader751: empty journal: appending 0 entries 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - leader751: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:03,477 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] JournalRecovery - leader751: journal recovery completedin 96.81 μs with journalIndex=1 09:27:03,478 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader751: ignoring DeleteMessagesSuccess(1) 09:27:03,952 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:27:03,952 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - leader751 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.247:17355, Removed) 09:27:03,952 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - leader751 (Follower): Received ElectionTimeout - switching to Candidate 09:27:03,952 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader751 (Candidate): Election: Candidate has following voting peers: [follower753, follower752] 09:27:03,956 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=6, votedFor=leader751 09:27:03,956 AM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - leader751 (Candidate): Starting new election term 6 09:27:03,956 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - leader751 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 09:27:04,442 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader751 (Candidate): Received ElectionTimeout 09:27:04,446 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=7, votedFor=leader751 09:27:04,446 AM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - leader751 (Candidate): Starting new election term 7 09:27:04,922 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader751 (Candidate): Received ElectionTimeout 09:27:04,925 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=8, votedFor=leader751 09:27:04,925 AM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - leader751 (Candidate): Starting new election term 8 09:27:05,352 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader751 (Candidate): Received ElectionTimeout 09:27:05,356 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=9, votedFor=leader751 09:27:05,356 AM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader751 (Candidate): Starting new election term 9 09:27:05,792 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader751 (Candidate): Received ElectionTimeout 09:27:05,796 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=10, votedFor=leader751 09:27:05,796 AM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - leader751 (Candidate): Starting new election term 10 09:27:06,302 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader751 (Candidate): Received ElectionTimeout 09:27:06,305 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=11, votedFor=leader751 09:27:06,305 AM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - leader751 (Candidate): Starting new election term 11 09:27:06,802 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader751 (Candidate): Received ElectionTimeout 09:27:06,805 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=12, votedFor=leader751 09:27:06,805 AM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader751 (Candidate): Starting new election term 12 09:27:07,242 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader751 (Candidate): Received ElectionTimeout 09:27:07,245 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=13, votedFor=leader751 09:27:07,245 AM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader751 (Candidate): Starting new election term 13 09:27:07,692 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader751 (Candidate): Received ElectionTimeout 09:27:07,697 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=14, votedFor=leader751 09:27:07,697 AM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - leader751 (Candidate): Starting new election term 14 09:27:08,182 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader751 (Candidate): Received ElectionTimeout 09:27:08,185 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader751: Set currentTerm=15, votedFor=leader751 09:27:08,185 AM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader751 (Candidate): Starting new election term 15 09:27:08,559 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower752-collector754#1570907821] 09:27:08,571 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower752] 09:27:08,571 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower752: terminating thread VirtualThread[#813,follower752-writer-148]/waiting 09:27:08,571 AM [follower752-writer-148] [DEBUG] JournalWriteTask - follower752: received 1 commands after 5.109 s 09:27:08,572 AM [follower752-writer-148] [DEBUG] JournalWriteTask - follower752: completed 1 commands in 47.92 μs 09:27:08,572 AM [follower752-writer-148] [DEBUG] JournalWriteTask - follower752: journal writer stopped 09:27:08,572 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower752: journal closed 09:27:08,572 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower752: journal closed 09:27:08,572 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower752: stopped executor with 0 remaining tasks 09:27:08,572 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower752: stopped executor with 0 remaining tasks 09:27:08,572 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower753-collector755#-745558334] 09:27:08,572 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower753] 09:27:08,572 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower753: terminating thread VirtualThread[#816,follower753-writer-149]/waiting 09:27:08,572 AM [follower753-writer-149] [DEBUG] JournalWriteTask - follower753: received 1 commands after 5.107 s 09:27:08,573 AM [follower753-writer-149] [DEBUG] JournalWriteTask - follower753: completed 1 commands in 18.23 μs 09:27:08,573 AM [follower753-writer-149] [DEBUG] JournalWriteTask - follower753: journal writer stopped 09:27:08,573 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower753: journal closed 09:27:08,573 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower753: journal closed 09:27:08,573 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower753: stopped executor with 0 remaining tasks 09:27:08,573 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower753: stopped executor with 0 remaining tasks 09:27:08,573 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader751-collector756#14417097] 09:27:08,573 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader751] 09:27:08,573 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - leader751: terminating thread VirtualThread[#818,leader751-writer-150]/waiting 09:27:08,573 AM [leader751-writer-150] [DEBUG] JournalWriteTask - leader751: received 1 commands after 5.103 s 09:27:08,573 AM [leader751-writer-150] [DEBUG] JournalWriteTask - leader751: completed 1 commands in 15.66 μs 09:27:08,573 AM [leader751-writer-150] [DEBUG] JournalWriteTask - leader751: journal writer stopped 09:27:08,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader751: journal closed 09:27:08,574 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader751: journal closed 09:27:08,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader751: stopped executor with 0 remaining tasks 09:27:08,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader751: stopped executor with 0 remaining tasks 09:27:08,578 AM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaInstallSnapshot starting 09:27:08,580 AM [test-pekko.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower761] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [07/05/2025 09:27:08.580] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower761]: org.opendaylight.controller.cluster.common.actor.Monitor@5fc20c [INFO] [pekkoDeadLetter][07/05/2025 09:27:08.580] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower761] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:08,581 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower761: Starting recovery with journal batch size 1000 09:27:08,581 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower761: started executor 09:27:08,581 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower761: started executor 09:27:08,581 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower761: metadata journal open: replayFrom=1 applyTo=0 09:27:08,582 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower761: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:08,582 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower761: journal open: applyTo=0 09:27:08,583 AM [follower761-writer-151] [DEBUG] JournalWriteTask - follower761: journal writer started 09:27:08,583 AM [follower761-writer-151] [DEBUG] JournalWriteTask - follower761: waiting to receive commands [WARN] [07/05/2025 09:27:08.583] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower762]: org.opendaylight.controller.cluster.common.actor.Monitor@51100dd5 [INFO] [pekkoDeadLetter][07/05/2025 09:27:08.583] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower762] to Actor[pekko://test/user/termination-monitor] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:08,583 AM [test-pekko.actor.default-dispatcher-15] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower762] to Actor[pekko://test/user/termination-monitor] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:08,583 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower762: Starting recovery with journal batch size 1000 09:27:08,583 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower762: started executor 09:27:08,584 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/follower761/TermInfo.properties does not exist 09:27:08,584 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower762: started executor 09:27:08,584 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower761: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/follower761/journal-v1-1.log 09:27:08,584 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower761: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/follower761/metadata-v1-1.log 09:27:08,584 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower761: no eligible files found 09:27:08,584 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower761: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:27:08,584 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower762: metadata journal open: replayFrom=1 applyTo=0 09:27:08,585 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower762: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:08,585 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower762: journal open: applyTo=0 09:27:08,585 AM [follower762-writer-152] [DEBUG] JournalWriteTask - follower762: journal writer started 09:27:08,585 AM [follower762-writer-152] [DEBUG] JournalWriteTask - follower762: waiting to receive commands 09:27:08,586 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/follower762/TermInfo.properties does not exist 09:27:08,587 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower762: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/follower762/journal-v1-1.log 09:27:08,587 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower762: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/follower762/metadata-v1-1.log 09:27:08,587 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower762: no eligible files found 09:27:08,587 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower762: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/05/2025 09:27:08.587] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader760]: org.opendaylight.controller.cluster.common.actor.Monitor@64904b73 [INFO] [pekkoDeadLetter][07/05/2025 09:27:08.587] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader760] to Actor[pekko://test/user/termination-monitor] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:08,587 AM [test-pekko.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader760] to Actor[pekko://test/user/termination-monitor] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:08,587 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - leader760: Starting recovery with journal batch size 1000 09:27:08,587 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader760: started executor 09:27:08,587 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader760: started executor 09:27:08,588 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader760: metadata journal open: replayFrom=1 applyTo=0 09:27:08,588 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader760: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:08,588 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader760: journal open: applyTo=0 09:27:08,589 AM [leader760-writer-153] [DEBUG] JournalWriteTask - leader760: journal writer started 09:27:08,589 AM [leader760-writer-153] [DEBUG] JournalWriteTask - leader760: waiting to receive commands 09:27:08,589 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/leader760/TermInfo.properties does not exist 09:27:08,589 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/leader760/journal-v1-1.log 09:27:08,589 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/leader760/metadata-v1-1.log 09:27:08,589 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader760: no eligible files found 09:27:08,589 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=5, votedFor=leader760 09:27:08,590 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader760: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:27:08,592 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower761: Set currentTerm=0, votedFor=null 09:27:08,592 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower761: Local TermInfo store seeded with TermInfo{term=0} 09:27:08,592 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower761: Pekko recovery completed and can restore from snapshot 09:27:08,592 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower761: starting journal recovery 09:27:08,592 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower761: Moving last applied index from -1 to -1 09:27:08,592 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower761: empty journal: appending 0 entries 09:27:08,592 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower761: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:08,593 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower761: journal recovery completedin 116.6 μs with journalIndex=1 09:27:08,596 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower762: Set currentTerm=0, votedFor=null 09:27:08,596 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=5, votedFor=leader760 09:27:08,596 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower762: Local TermInfo store seeded with TermInfo{term=0} 09:27:08,596 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader760: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader760} 09:27:08,596 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower762: Pekko recovery completed and can restore from snapshot 09:27:08,596 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader760: taking snapshot to clear Pekko persistence to 1 09:27:08,597 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Recovery - leader760: no applied entries in recovery log, re-snapshotting EntryInfo[index=-1, term=-1] 09:27:08,597 AM [test-pekko.actor.default-dispatcher-17] [INFO] Recovery - leader760: Taking snapshot on entry with index -1 09:27:08,597 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower762: starting journal recovery 09:27:08,597 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower762: Moving last applied index from -1 to -1 09:27:08,597 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader760: support() called 09:27:08,597 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower762: empty journal: appending 0 entries 09:27:08,597 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower762: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:08,597 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower762: journal recovery completedin 286.2 μs with journalIndex=1 09:27:08,597 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader760: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/leader760/snapshot-000000006868EFEC-239DD18C.tmp 09:27:08,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader760: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/leader760/snapshot-000000006868EFEC-239DD18C.v1 09:27:08,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader760: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/leader760/snapshot-000000006868EFEC-239DD18C.v1 09:27:08,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/leader760/TermInfo.properties 09:27:08,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/leader760/journal-v1-1.log 09:27:08,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader760: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/leader760/metadata-v1-1.log 09:27:08,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader760: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-05T09:27:08.597545356Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest16618287622103146772/leader760/snapshot-000000006868EFEC-239DD18C.v1, size=221, entryCompress=NONE, stateCompress=NONE} 09:27:08,601 AM [test-pekko.actor.default-dispatcher-17] [INFO] Recovery - leader760: Snapshot completed in 4.080 ms, resetting timer for the next recovery snapshot 09:27:08,601 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader760: Pekko recovery completed and can restore from snapshot 09:27:08,601 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader760: starting journal recovery 09:27:08,601 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader760: Moving last applied index from -1 to -1 09:27:08,601 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - leader760: empty journal: appending 0 entries 09:27:08,601 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader760: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:08,601 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - leader760: journal recovery completedin 168.3 μs with journalIndex=1 09:27:08,601 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader760: ignoring DeleteMessagesSuccess(1) 09:27:09,62 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:27:09,62 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader760 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.247:17355, Removed) 09:27:09,62 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader760 (Follower): Received ElectionTimeout - switching to Candidate 09:27:09,62 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader760 (Candidate): Election: Candidate has following voting peers: [follower762, follower761] 09:27:09,65 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=6, votedFor=leader760 09:27:09,65 AM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader760 (Candidate): Starting new election term 6 09:27:09,65 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader760 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 09:27:09,512 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader760 (Candidate): Received ElectionTimeout 09:27:09,516 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=7, votedFor=leader760 09:27:09,516 AM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader760 (Candidate): Starting new election term 7 09:27:10,12 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader760 (Candidate): Received ElectionTimeout 09:27:10,16 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=8, votedFor=leader760 09:27:10,16 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - leader760 (Candidate): Starting new election term 8 09:27:10,492 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader760 (Candidate): Received ElectionTimeout 09:27:10,495 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=9, votedFor=leader760 09:27:10,496 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - leader760 (Candidate): Starting new election term 9 09:27:11,12 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader760 (Candidate): Received ElectionTimeout 09:27:11,16 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=10, votedFor=leader760 09:27:11,16 AM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader760 (Candidate): Starting new election term 10 09:27:11,522 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader760 (Candidate): Received ElectionTimeout 09:27:11,527 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=11, votedFor=leader760 09:27:11,527 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - leader760 (Candidate): Starting new election term 11 09:27:11,982 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader760 (Candidate): Received ElectionTimeout 09:27:11,985 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=12, votedFor=leader760 09:27:11,985 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - leader760 (Candidate): Starting new election term 12 09:27:12,462 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader760 (Candidate): Received ElectionTimeout 09:27:12,465 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=13, votedFor=leader760 09:27:12,466 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - leader760 (Candidate): Starting new election term 13 09:27:12,912 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader760 (Candidate): Received ElectionTimeout 09:27:12,915 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=14, votedFor=leader760 09:27:12,915 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - leader760 (Candidate): Starting new election term 14 09:27:13,352 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader760 (Candidate): Received ElectionTimeout 09:27:13,356 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader760: Set currentTerm=15, votedFor=leader760 09:27:13,356 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - leader760 (Candidate): Starting new election term 15 09:27:13,634 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower761-collector763#1603972124] 09:27:13,635 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower761] 09:27:13,635 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower761: terminating thread VirtualThread[#819,follower761-writer-151]/waiting 09:27:13,635 AM [follower761-writer-151] [DEBUG] JournalWriteTask - follower761: received 1 commands after 5.053 s 09:27:13,635 AM [follower761-writer-151] [DEBUG] JournalWriteTask - follower761: completed 1 commands in 34.50 μs 09:27:13,636 AM [follower761-writer-151] [DEBUG] JournalWriteTask - follower761: journal writer stopped 09:27:13,636 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower761: journal closed 09:27:13,636 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower761: journal closed 09:27:13,636 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower761: stopped executor with 0 remaining tasks 09:27:13,636 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower761: stopped executor with 0 remaining tasks 09:27:13,636 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower762-collector764#-1020847675] 09:27:13,636 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower762] 09:27:13,636 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower762: terminating thread VirtualThread[#820,follower762-writer-152]/waiting 09:27:13,636 AM [follower762-writer-152] [DEBUG] JournalWriteTask - follower762: received 1 commands after 5.052 s 09:27:13,636 AM [follower762-writer-152] [DEBUG] JournalWriteTask - follower762: completed 1 commands in 17.20 μs 09:27:13,637 AM [follower762-writer-152] [DEBUG] JournalWriteTask - follower762: journal writer stopped 09:27:13,637 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower762: journal closed 09:27:13,637 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower762: journal closed 09:27:13,637 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower762: stopped executor with 0 remaining tasks 09:27:13,637 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower762: stopped executor with 0 remaining tasks 09:27:13,637 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader760-collector765#424950338] 09:27:13,637 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader760] 09:27:13,637 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - leader760: terminating thread VirtualThread[#823,leader760-writer-153]/waiting 09:27:13,637 AM [leader760-writer-153] [DEBUG] JournalWriteTask - leader760: received 1 commands after 5.049 s 09:27:13,637 AM [leader760-writer-153] [DEBUG] JournalWriteTask - leader760: completed 1 commands in 16.83 μs 09:27:13,637 AM [leader760-writer-153] [DEBUG] JournalWriteTask - leader760: journal writer stopped 09:27:13,637 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader760: journal closed 09:27:13,637 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader760: journal closed 09:27:13,637 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader760: stopped executor with 0 remaining tasks 09:27:13,637 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader760: stopped executor with 0 remaining tasks 09:27:13,640 AM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotWithLaggingFollowerCaughtUpViaAppendEntries starting 09:27:13,642 AM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower770] to Actor[pekko://test/user/termination-monitor] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [07/05/2025 09:27:13.642] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower770]: org.opendaylight.controller.cluster.common.actor.Monitor@13d80eb [INFO] [pekkoDeadLetter][07/05/2025 09:27:13.641] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower770] to Actor[pekko://test/user/termination-monitor] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:13,642 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower770: Starting recovery with journal batch size 1000 09:27:13,642 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower770: started executor 09:27:13,642 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower770: started executor 09:27:13,642 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower770: metadata journal open: replayFrom=1 applyTo=0 09:27:13,643 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower770: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:13,643 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower770: journal open: applyTo=0 09:27:13,643 AM [follower770-writer-154] [DEBUG] JournalWriteTask - follower770: journal writer started 09:27:13,643 AM [follower770-writer-154] [DEBUG] JournalWriteTask - follower770: waiting to receive commands [WARN] [07/05/2025 09:27:13.644] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower771]: org.opendaylight.controller.cluster.common.actor.Monitor@67326f3d 09:27:13,644 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/follower770/TermInfo.properties does not exist [INFO] [pekkoDeadLetter][07/05/2025 09:27:13.644] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower771] to Actor[pekko://test/user/termination-monitor] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:13,644 AM [test-pekko.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower771] to Actor[pekko://test/user/termination-monitor] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:13,644 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower770: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/follower770/journal-v1-1.log 09:27:13,644 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower770: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/follower770/metadata-v1-1.log 09:27:13,644 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower770: no eligible files found 09:27:13,644 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower771: Starting recovery with journal batch size 1000 09:27:13,644 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower770: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:27:13,644 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower771: started executor 09:27:13,644 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower771: started executor 09:27:13,645 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower771: metadata journal open: replayFrom=1 applyTo=0 09:27:13,645 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower771: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:13,645 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower771: journal open: applyTo=0 09:27:13,645 AM [follower771-writer-155] [DEBUG] JournalWriteTask - follower771: journal writer started 09:27:13,645 AM [follower771-writer-155] [DEBUG] JournalWriteTask - follower771: waiting to receive commands 09:27:13,646 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/follower771/TermInfo.properties does not exist 09:27:13,646 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/follower771/journal-v1-1.log 09:27:13,646 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower771: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/follower771/metadata-v1-1.log 09:27:13,646 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower771: no eligible files found 09:27:13,646 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower771: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/05/2025 09:27:13.646] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader769]: org.opendaylight.controller.cluster.common.actor.Monitor@6bdc1374 09:27:13,646 AM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader769] to Actor[pekko://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][07/05/2025 09:27:13.646] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader769] to Actor[pekko://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:13,647 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - leader769: Starting recovery with journal batch size 1000 09:27:13,647 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower770: Set currentTerm=0, votedFor=null 09:27:13,647 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower770: Local TermInfo store seeded with TermInfo{term=0} 09:27:13,647 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader769: started executor 09:27:13,647 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower770: Pekko recovery completed and can restore from snapshot 09:27:13,648 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower770: starting journal recovery 09:27:13,648 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower770: Moving last applied index from -1 to -1 09:27:13,648 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader769: started executor 09:27:13,648 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower770: empty journal: appending 0 entries 09:27:13,648 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower770: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:13,648 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower770: journal recovery completedin 220.5 μs with journalIndex=1 09:27:13,648 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader769: metadata journal open: replayFrom=1 applyTo=0 09:27:13,648 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower771: Set currentTerm=0, votedFor=null 09:27:13,648 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower771: Local TermInfo store seeded with TermInfo{term=0} 09:27:13,649 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower771: Pekko recovery completed and can restore from snapshot 09:27:13,649 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower771: starting journal recovery 09:27:13,649 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower771: Moving last applied index from -1 to -1 09:27:13,649 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower771: empty journal: appending 0 entries 09:27:13,649 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower771: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:13,649 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader769: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:13,649 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - leader769: journal open: applyTo=0 09:27:13,649 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower771: journal recovery completedin 117.7 μs with journalIndex=1 09:27:13,649 AM [leader769-writer-156] [DEBUG] JournalWriteTask - leader769: journal writer started 09:27:13,649 AM [leader769-writer-156] [DEBUG] JournalWriteTask - leader769: waiting to receive commands 09:27:13,650 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/leader769/TermInfo.properties does not exist 09:27:13,650 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/leader769/journal-v1-1.log 09:27:13,650 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/leader769/metadata-v1-1.log 09:27:13,650 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: no eligible files found 09:27:13,650 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=5, votedFor=leader769 09:27:13,650 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader769: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:27:13,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=5, votedFor=leader769 09:27:13,656 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader769: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader769} 09:27:13,656 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader769: taking snapshot to clear Pekko persistence to 1 09:27:13,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Recovery - leader769: no applied entries in recovery log, re-snapshotting EntryInfo[index=-1, term=-1] 09:27:13,656 AM [test-pekko.actor.default-dispatcher-6] [INFO] Recovery - leader769: Taking snapshot on entry with index -1 09:27:13,656 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader769: support() called 09:27:13,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/leader769/snapshot-000000006868EFF1-2724529A.tmp 09:27:13,658 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/leader769/snapshot-000000006868EFF1-2724529A.v1 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/leader769/snapshot-000000006868EFF1-2724529A.v1 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/leader769/TermInfo.properties 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/leader769/journal-v1-1.log 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/leader769/metadata-v1-1.log 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader769: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-05T09:27:13.656691866Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest9102096838767631159/leader769/snapshot-000000006868EFF1-2724529A.v1, size=221, entryCompress=NONE, stateCompress=NONE} 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [INFO] Recovery - leader769: Snapshot completed in 2.942 ms, resetting timer for the next recovery snapshot 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: Pekko recovery completed and can restore from snapshot 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: starting journal recovery 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader769: Moving last applied index from -1 to -1 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader769: empty journal: appending 0 entries 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader769: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader769: journal recovery completedin 130.9 μs with journalIndex=1 09:27:13,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader769: ignoring DeleteMessagesSuccess(1) 09:27:14,152 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:27:14,152 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader769 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.247:17355, Removed) 09:27:14,152 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader769 (Follower): Received ElectionTimeout - switching to Candidate 09:27:14,152 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader769 (Candidate): Election: Candidate has following voting peers: [follower771, follower770] 09:27:14,155 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=6, votedFor=leader769 09:27:14,155 AM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader769 (Candidate): Starting new election term 6 09:27:14,155 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader769 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 09:27:14,642 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader769 (Candidate): Received ElectionTimeout 09:27:14,646 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=7, votedFor=leader769 09:27:14,646 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader769 (Candidate): Starting new election term 7 09:27:15,122 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader769 (Candidate): Received ElectionTimeout 09:27:15,126 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=8, votedFor=leader769 09:27:15,126 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader769 (Candidate): Starting new election term 8 09:27:15,602 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader769 (Candidate): Received ElectionTimeout 09:27:15,605 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=9, votedFor=leader769 09:27:15,605 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader769 (Candidate): Starting new election term 9 09:27:16,82 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader769 (Candidate): Received ElectionTimeout 09:27:16,86 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=10, votedFor=leader769 09:27:16,86 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader769 (Candidate): Starting new election term 10 09:27:16,511 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader769 (Candidate): Received ElectionTimeout 09:27:16,515 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=11, votedFor=leader769 09:27:16,515 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader769 (Candidate): Starting new election term 11 09:27:17,11 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader769 (Candidate): Received ElectionTimeout 09:27:17,15 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=12, votedFor=leader769 09:27:17,15 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader769 (Candidate): Starting new election term 12 09:27:17,482 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader769 (Candidate): Received ElectionTimeout 09:27:17,487 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=13, votedFor=leader769 09:27:17,487 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader769 (Candidate): Starting new election term 13 09:27:17,941 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader769 (Candidate): Received ElectionTimeout 09:27:17,948 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=14, votedFor=leader769 09:27:17,948 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader769 (Candidate): Starting new election term 14 09:27:18,382 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader769 (Candidate): Received ElectionTimeout 09:27:18,385 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader769: Set currentTerm=15, votedFor=leader769 09:27:18,385 AM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader769 (Candidate): Starting new election term 15 09:27:18,695 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower770-collector772#-1615840714] 09:27:18,707 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower770] 09:27:18,707 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower770: terminating thread VirtualThread[#824,follower770-writer-154]/waiting 09:27:18,707 AM [follower770-writer-154] [DEBUG] JournalWriteTask - follower770: received 1 commands after 5.064 s 09:27:18,707 AM [follower770-writer-154] [DEBUG] JournalWriteTask - follower770: completed 1 commands in 41.77 μs 09:27:18,707 AM [follower770-writer-154] [DEBUG] JournalWriteTask - follower770: journal writer stopped 09:27:18,708 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower770: journal closed 09:27:18,708 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower770: journal closed 09:27:18,708 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower770: stopped executor with 0 remaining tasks 09:27:18,708 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower770: stopped executor with 0 remaining tasks 09:27:18,708 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower771-collector773#-744363708] 09:27:18,708 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower771] 09:27:18,708 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower771: terminating thread VirtualThread[#825,follower771-writer-155]/waiting 09:27:18,708 AM [follower771-writer-155] [DEBUG] JournalWriteTask - follower771: received 1 commands after 5.063 s 09:27:18,708 AM [follower771-writer-155] [DEBUG] JournalWriteTask - follower771: completed 1 commands in 18.07 μs 09:27:18,708 AM [follower771-writer-155] [DEBUG] JournalWriteTask - follower771: journal writer stopped 09:27:18,709 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower771: journal closed 09:27:18,709 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower771: journal closed 09:27:18,709 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower771: stopped executor with 0 remaining tasks 09:27:18,709 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower771: stopped executor with 0 remaining tasks 09:27:18,709 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader769-collector774#-1917911924] 09:27:18,709 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader769] 09:27:18,709 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - leader769: terminating thread VirtualThread[#826,leader769-writer-156]/waiting 09:27:18,709 AM [leader769-writer-156] [DEBUG] JournalWriteTask - leader769: received 1 commands after 5.060 s 09:27:18,709 AM [leader769-writer-156] [DEBUG] JournalWriteTask - leader769: completed 1 commands in 17.29 μs 09:27:18,709 AM [leader769-writer-156] [DEBUG] JournalWriteTask - leader769: journal writer stopped 09:27:18,709 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader769: journal closed 09:27:18,710 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - leader769: journal closed 09:27:18,710 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader769: stopped executor with 0 remaining tasks 09:27:18,710 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader769: stopped executor with 0 remaining tasks 09:27:18,713 AM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderInstallsSnapshotWithRestartedFollowerDuringSnapshotInstallation starting 09:27:18,715 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower779: Starting recovery with journal batch size 1000 09:27:18,715 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower779: started executor 09:27:18,715 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower779: started executor 09:27:18,716 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower779: metadata journal open: replayFrom=1 applyTo=0 [WARN] [07/05/2025 09:27:18.716] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower779]: org.opendaylight.controller.cluster.common.actor.Monitor@33280988 09:27:18,716 AM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower779] to Actor[pekko://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:18,716 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower779: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:18,716 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower779: journal open: applyTo=0 [INFO] [pekkoDeadLetter][07/05/2025 09:27:18.716] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower779] to Actor[pekko://test/user/termination-monitor] was not delivered. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:27:18,716 AM [follower779-writer-157] [DEBUG] JournalWriteTask - follower779: journal writer started 09:27:18,716 AM [follower779-writer-157] [DEBUG] JournalWriteTask - follower779: waiting to receive commands 09:27:18,717 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/follower779/TermInfo.properties does not exist 09:27:18,717 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower779: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/follower779/journal-v1-1.log 09:27:18,717 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower779: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/follower779/metadata-v1-1.log 09:27:18,718 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower779: no eligible files found 09:27:18,718 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower779: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/05/2025 09:27:18.718] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower780]: org.opendaylight.controller.cluster.common.actor.Monitor@7c336e98 09:27:18,718 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower780: Starting recovery with journal batch size 1000 09:27:18,718 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower780: started executor 09:27:18,719 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower780: started executor 09:27:18,719 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower780: metadata journal open: replayFrom=1 applyTo=0 09:27:18,720 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower780: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:18,720 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower780: journal open: applyTo=0 09:27:18,720 AM [follower780-writer-158] [DEBUG] JournalWriteTask - follower780: journal writer started 09:27:18,720 AM [follower780-writer-158] [DEBUG] JournalWriteTask - follower780: waiting to receive commands 09:27:18,720 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/follower780/TermInfo.properties does not exist 09:27:18,720 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower780: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/follower780/journal-v1-1.log 09:27:18,720 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower780: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/follower780/metadata-v1-1.log 09:27:18,720 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower780: no eligible files found 09:27:18,720 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower780: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/05/2025 09:27:18.721] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader778]: org.opendaylight.controller.cluster.common.actor.Monitor@3ec8f9be 09:27:18,722 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower779: Set currentTerm=0, votedFor=null 09:27:18,722 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower779: Local TermInfo store seeded with TermInfo{term=0} 09:27:18,722 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower779: Pekko recovery completed and can restore from snapshot 09:27:18,722 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower779: starting journal recovery 09:27:18,722 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower779: Moving last applied index from -1 to -1 09:27:18,722 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower779: empty journal: appending 0 entries 09:27:18,722 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower779: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:18,722 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower779: journal recovery completedin 157.5 μs with journalIndex=1 09:27:18,722 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - leader778: Starting recovery with journal batch size 1000 09:27:18,722 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader778: started executor 09:27:18,722 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader778: started executor 09:27:18,722 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader778: metadata journal open: replayFrom=1 applyTo=0 09:27:18,723 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower780: Set currentTerm=0, votedFor=null 09:27:18,723 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower780: Local TermInfo store seeded with TermInfo{term=0} 09:27:18,723 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower780: Pekko recovery completed and can restore from snapshot 09:27:18,723 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower780: starting journal recovery 09:27:18,723 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower780: Moving last applied index from -1 to -1 09:27:18,723 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower780: empty journal: appending 0 entries 09:27:18,723 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower780: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:18,723 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower780: journal recovery completedin 135.5 μs with journalIndex=1 09:27:18,723 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader778: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:18,723 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader778: journal open: applyTo=0 09:27:18,723 AM [leader778-writer-159] [DEBUG] JournalWriteTask - leader778: journal writer started 09:27:18,723 AM [leader778-writer-159] [DEBUG] JournalWriteTask - leader778: waiting to receive commands 09:27:18,724 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/leader778/TermInfo.properties does not exist 09:27:18,724 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/leader778/journal-v1-1.log 09:27:18,724 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/leader778/metadata-v1-1.log 09:27:18,724 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: no eligible files found 09:27:18,724 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=5, votedFor=leader778 09:27:18,724 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - leader778: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:27:18,727 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=5, votedFor=leader778 09:27:18,727 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - leader778: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader778} 09:27:18,727 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - leader778: taking snapshot to clear Pekko persistence to 1 09:27:18,727 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Recovery - leader778: no applied entries in recovery log, re-snapshotting EntryInfo[index=-1, term=-1] 09:27:18,727 AM [test-pekko.actor.default-dispatcher-14] [INFO] Recovery - leader778: Taking snapshot on entry with index -1 09:27:18,727 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader778: support() called 09:27:18,727 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/leader778/snapshot-000000006868EFF6-2B5E86BA.tmp 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/leader778/snapshot-000000006868EFF6-2B5E86BA.v1 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/leader778/snapshot-000000006868EFF6-2B5E86BA.v1 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/leader778/TermInfo.properties 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/leader778/journal-v1-1.log 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/leader778/metadata-v1-1.log 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-05T09:27:18.727615162Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest14114628172524005463/leader778/snapshot-000000006868EFF6-2B5E86BA.v1, size=221, entryCompress=NONE, stateCompress=NONE} 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [INFO] Recovery - leader778: Snapshot completed in 2.404 ms, resetting timer for the next recovery snapshot 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader778: Pekko recovery completed and can restore from snapshot 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader778: starting journal recovery 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader778: Moving last applied index from -1 to -1 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] JournalRecovery - leader778: empty journal: appending 0 entries 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - leader778: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:18,729 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] JournalRecovery - leader778: journal recovery completedin 172.7 μs with journalIndex=1 09:27:18,730 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader778: ignoring DeleteMessagesSuccess(1) 09:27:19,172 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:27:19,172 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader778 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.247:17355, Removed) 09:27:19,173 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader778 (Follower): Received ElectionTimeout - switching to Candidate 09:27:19,173 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader778 (Candidate): Election: Candidate has following voting peers: [follower779, follower780] 09:27:19,176 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=6, votedFor=leader778 09:27:19,176 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader778 (Candidate): Starting new election term 6 09:27:19,176 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActorBehavior - leader778 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 09:27:19,681 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader778 (Candidate): Received ElectionTimeout 09:27:19,691 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=7, votedFor=leader778 09:27:19,691 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader778 (Candidate): Starting new election term 7 09:27:20,132 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader778 (Candidate): Received ElectionTimeout 09:27:20,136 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=8, votedFor=leader778 09:27:20,136 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader778 (Candidate): Starting new election term 8 09:27:20,592 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader778 (Candidate): Received ElectionTimeout 09:27:20,596 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=9, votedFor=leader778 09:27:20,596 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader778 (Candidate): Starting new election term 9 09:27:21,102 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader778 (Candidate): Received ElectionTimeout 09:27:21,106 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=10, votedFor=leader778 09:27:21,106 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader778 (Candidate): Starting new election term 10 09:27:21,562 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader778 (Candidate): Received ElectionTimeout 09:27:21,566 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=11, votedFor=leader778 09:27:21,566 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader778 (Candidate): Starting new election term 11 09:27:22,62 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader778 (Candidate): Received ElectionTimeout 09:27:22,66 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=12, votedFor=leader778 09:27:22,66 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader778 (Candidate): Starting new election term 12 09:27:22,542 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader778 (Candidate): Received ElectionTimeout 09:27:22,545 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=13, votedFor=leader778 09:27:22,545 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader778 (Candidate): Starting new election term 13 09:27:22,972 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader778 (Candidate): Received ElectionTimeout 09:27:22,976 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=14, votedFor=leader778 09:27:22,976 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader778 (Candidate): Starting new election term 14 09:27:23,492 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader778 (Candidate): Received ElectionTimeout 09:27:23,496 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader778: Set currentTerm=15, votedFor=leader778 09:27:23,496 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader778 (Candidate): Starting new election term 15 09:27:23,764 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower779-collector781#1474485477] 09:27:23,775 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower779] 09:27:23,775 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower779: terminating thread VirtualThread[#827,follower779-writer-157]/waiting 09:27:23,775 AM [follower779-writer-157] [DEBUG] JournalWriteTask - follower779: received 1 commands after 5.059 s 09:27:23,775 AM [follower779-writer-157] [DEBUG] JournalWriteTask - follower779: completed 1 commands in 41.52 μs 09:27:23,775 AM [follower779-writer-157] [DEBUG] JournalWriteTask - follower779: journal writer stopped 09:27:23,776 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower779: journal closed 09:27:23,776 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower779: journal closed 09:27:23,776 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower779: stopped executor with 0 remaining tasks 09:27:23,776 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower779: stopped executor with 0 remaining tasks 09:27:23,776 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower780-collector782#1430960225] 09:27:23,776 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower780] 09:27:23,776 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower780: terminating thread VirtualThread[#828,follower780-writer-158]/waiting 09:27:23,777 AM [follower780-writer-158] [DEBUG] JournalWriteTask - follower780: received 1 commands after 5.057 s 09:27:23,777 AM [follower780-writer-158] [DEBUG] JournalWriteTask - follower780: completed 1 commands in 19.19 μs 09:27:23,777 AM [follower780-writer-158] [DEBUG] JournalWriteTask - follower780: journal writer stopped 09:27:23,777 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower780: journal closed 09:27:23,777 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower780: journal closed 09:27:23,777 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower780: stopped executor with 0 remaining tasks 09:27:23,777 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower780: stopped executor with 0 remaining tasks 09:27:23,777 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader778-collector783#671411988] 09:27:23,777 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader778] 09:27:23,777 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PekkoRaftStorage - leader778: terminating thread VirtualThread[#829,leader778-writer-159]/waiting 09:27:23,778 AM [leader778-writer-159] [DEBUG] JournalWriteTask - leader778: received 1 commands after 5.055 s 09:27:23,778 AM [leader778-writer-159] [DEBUG] JournalWriteTask - leader778: completed 1 commands in 19.09 μs 09:27:23,778 AM [leader778-writer-159] [DEBUG] JournalWriteTask - leader778: journal writer stopped 09:27:23,778 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - leader778: journal closed 09:27:23,778 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRaftStorage - leader778: journal closed 09:27:23,778 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: stopped executor with 0 remaining tasks 09:27:23,778 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader778: stopped executor with 0 remaining tasks 09:27:23,781 AM [main] [INFO] ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest - testLeaderSnapshotTriggeredByMemoryThresholdExceededWithLaggingFollower starting 09:27:23,783 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActor - follower788: Starting recovery with journal batch size 1000 09:27:23,783 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower788: started executor 09:27:23,783 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower788: started executor 09:27:23,784 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - follower788: metadata journal open: replayFrom=1 applyTo=0 09:27:23,784 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - follower788: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:23,784 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRaftStorage - follower788: journal open: applyTo=0 [WARN] [07/05/2025 09:27:23.784] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower788]: org.opendaylight.controller.cluster.common.actor.Monitor@559991cd 09:27:23,784 AM [follower788-writer-160] [DEBUG] JournalWriteTask - follower788: journal writer started 09:27:23,784 AM [follower788-writer-160] [DEBUG] JournalWriteTask - follower788: waiting to receive commands 09:27:23,785 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/follower788/TermInfo.properties does not exist 09:27:23,785 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower788: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/follower788/journal-v1-1.log 09:27:23,785 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower788: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/follower788/metadata-v1-1.log 09:27:23,785 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower788: no eligible files found 09:27:23,785 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower788: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/05/2025 09:27:23.786] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower789]: org.opendaylight.controller.cluster.common.actor.Monitor@d8a0480 09:27:23,787 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower789: Starting recovery with journal batch size 1000 09:27:23,787 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: started executor 09:27:23,787 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower789: started executor 09:27:23,788 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower788: Set currentTerm=0, votedFor=null 09:27:23,788 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower789: metadata journal open: replayFrom=1 applyTo=0 09:27:23,788 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower788: Local TermInfo store seeded with TermInfo{term=0} 09:27:23,788 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower788: Pekko recovery completed and can restore from snapshot 09:27:23,788 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower788: starting journal recovery 09:27:23,788 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower788: Moving last applied index from -1 to -1 09:27:23,788 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower788: empty journal: appending 0 entries 09:27:23,788 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower788: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:23,788 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower788: journal recovery completedin 165.8 μs with journalIndex=1 09:27:23,788 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower789: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:23,788 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower789: journal open: applyTo=0 09:27:23,789 AM [follower789-writer-161] [DEBUG] JournalWriteTask - follower789: journal writer started 09:27:23,789 AM [follower789-writer-161] [DEBUG] JournalWriteTask - follower789: waiting to receive commands 09:27:23,790 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/follower789/TermInfo.properties does not exist 09:27:23,790 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/follower789/journal-v1-1.log 09:27:23,790 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower789: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/follower789/metadata-v1-1.log 09:27:23,790 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower789: no eligible files found 09:27:23,790 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower789: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [07/05/2025 09:27:23.791] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader787]: org.opendaylight.controller.cluster.common.actor.Monitor@5bd5d1d0 09:27:23,792 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - leader787: Starting recovery with journal batch size 1000 09:27:23,792 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader787: started executor 09:27:23,792 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader787: started executor 09:27:23,792 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower789: Set currentTerm=0, votedFor=null 09:27:23,792 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower789: Local TermInfo store seeded with TermInfo{term=0} 09:27:23,792 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower789: Pekko recovery completed and can restore from snapshot 09:27:23,792 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower789: starting journal recovery 09:27:23,792 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower789: Moving last applied index from -1 to -1 09:27:23,792 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower789: empty journal: appending 0 entries 09:27:23,792 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower789: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:23,793 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader787: metadata journal open: replayFrom=1 applyTo=0 09:27:23,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower789: journal recovery completedin 204.0 μs with journalIndex=1 09:27:23,793 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader787: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:27:23,793 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - leader787: journal open: applyTo=0 09:27:23,793 AM [leader787-writer-162] [DEBUG] JournalWriteTask - leader787: journal writer started 09:27:23,793 AM [leader787-writer-162] [DEBUG] JournalWriteTask - leader787: waiting to receive commands 09:27:23,794 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/leader787/TermInfo.properties does not exist 09:27:23,795 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/leader787/journal-v1-1.log 09:27:23,795 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/leader787/metadata-v1-1.log 09:27:23,795 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: no eligible files found 09:27:23,795 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=5, votedFor=leader787 09:27:23,795 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - leader787: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:27:23,797 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=5, votedFor=leader787 09:27:23,797 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - leader787: Local TermInfo store seeded with TermInfo{term=5, votedFor=leader787} 09:27:23,797 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - leader787: taking snapshot to clear Pekko persistence to 1 09:27:23,797 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Recovery - leader787: no applied entries in recovery log, re-snapshotting EntryInfo[index=-1, term=-1] 09:27:23,797 AM [test-pekko.actor.default-dispatcher-14] [INFO] Recovery - leader787: Taking snapshot on entry with index -1 09:27:23,797 AM [test-pekko.actor.default-dispatcher-14] [INFO] MockRaftActor - leader787: support() called 09:27:23,797 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/leader787/snapshot-000000006868EFFB-2F8B8349.tmp 09:27:23,799 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/leader787/snapshot-000000006868EFFB-2F8B8349.v1 09:27:23,799 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/leader787/snapshot-000000006868EFFB-2F8B8349.v1 09:27:23,799 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/leader787/TermInfo.properties 09:27:23,799 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/leader787/journal-v1-1.log 09:27:23,799 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/leader787/metadata-v1-1.log 09:27:23,799 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-05T09:27:23.797672265Z, file=/tmp/org.opendaylight.controller.cluster.raft.ReplicationAndSnapshotsWithLaggingFollowerIntegrationTest5491303205391880200/leader787/snapshot-000000006868EFFB-2F8B8349.v1, size=221, entryCompress=NONE, stateCompress=NONE} 09:27:23,800 AM [test-pekko.actor.default-dispatcher-14] [INFO] Recovery - leader787: Snapshot completed in 2.676 ms, resetting timer for the next recovery snapshot 09:27:23,800 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader787: Pekko recovery completed and can restore from snapshot 09:27:23,800 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader787: starting journal recovery 09:27:23,800 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - leader787: Moving last applied index from -1 to -1 09:27:23,800 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] JournalRecovery - leader787: empty journal: appending 0 entries 09:27:23,800 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - leader787: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:27:23,800 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] JournalRecovery - leader787: journal recovery completedin 196.2 μs with journalIndex=1 09:27:23,800 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - leader787: ignoring DeleteMessagesSuccess(1) 09:27:24,222 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:27:24,222 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader787 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.171.247:17355, Removed) 09:27:24,222 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Follower - leader787 (Follower): Received ElectionTimeout - switching to Candidate 09:27:24,222 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader787 (Candidate): Election: Candidate has following voting peers: [follower788, follower789] 09:27:24,227 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=6, votedFor=leader787 09:27:24,227 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader787 (Candidate): Starting new election term 6 09:27:24,227 AM [test-pekko.actor.default-dispatcher-14] [INFO] RaftActorBehavior - leader787 (Follower) :- Switching from behavior Follower to Candidate, election term: 6 09:27:24,742 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader787 (Candidate): Received ElectionTimeout 09:27:24,745 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=7, votedFor=leader787 09:27:24,745 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader787 (Candidate): Starting new election term 7 09:27:25,212 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader787 (Candidate): Received ElectionTimeout 09:27:25,219 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=8, votedFor=leader787 09:27:25,219 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader787 (Candidate): Starting new election term 8 09:27:25,642 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader787 (Candidate): Received ElectionTimeout 09:27:25,646 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=9, votedFor=leader787 09:27:25,646 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader787 (Candidate): Starting new election term 9 09:27:26,112 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader787 (Candidate): Received ElectionTimeout 09:27:26,118 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=10, votedFor=leader787 09:27:26,118 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader787 (Candidate): Starting new election term 10 09:27:26,581 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader787 (Candidate): Received ElectionTimeout 09:27:26,586 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=11, votedFor=leader787 09:27:26,586 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader787 (Candidate): Starting new election term 11 09:27:27,82 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader787 (Candidate): Received ElectionTimeout 09:27:27,86 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=12, votedFor=leader787 09:27:27,86 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader787 (Candidate): Starting new election term 12 09:27:27,522 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader787 (Candidate): Received ElectionTimeout 09:27:27,527 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=13, votedFor=leader787 09:27:27,527 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader787 (Candidate): Starting new election term 13 09:27:28,42 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader787 (Candidate): Received ElectionTimeout 09:27:28,45 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=14, votedFor=leader787 09:27:28,45 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader787 (Candidate): Starting new election term 14 09:27:28,461 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] Candidate - leader787 (Candidate): Received ElectionTimeout 09:27:28,465 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - leader787: Set currentTerm=15, votedFor=leader787 09:27:28,465 AM [test-pekko.actor.default-dispatcher-14] [INFO] Candidate - leader787 (Candidate): Starting new election term 15 09:27:28,833 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower788-collector790#-417738646] 09:27:28,844 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower788] 09:27:28,844 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PekkoRaftStorage - follower788: terminating thread VirtualThread[#830,follower788-writer-160]/waiting 09:27:28,844 AM [follower788-writer-160] [DEBUG] JournalWriteTask - follower788: received 1 commands after 5.060 s 09:27:28,845 AM [follower788-writer-160] [DEBUG] JournalWriteTask - follower788: completed 1 commands in 30.84 μs 09:27:28,845 AM [follower788-writer-160] [DEBUG] JournalWriteTask - follower788: journal writer stopped 09:27:28,845 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - follower788: journal closed 09:27:28,845 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRaftStorage - follower788: journal closed 09:27:28,845 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower788: stopped executor with 0 remaining tasks 09:27:28,845 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower788: stopped executor with 0 remaining tasks 09:27:28,845 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower789-collector791#397479778] 09:27:28,845 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower789] 09:27:28,845 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PekkoRaftStorage - follower789: terminating thread VirtualThread[#831,follower789-writer-161]/waiting 09:27:28,845 AM [follower789-writer-161] [DEBUG] JournalWriteTask - follower789: received 1 commands after 5.057 s 09:27:28,845 AM [follower789-writer-161] [DEBUG] JournalWriteTask - follower789: completed 1 commands in 17.11 μs 09:27:28,845 AM [follower789-writer-161] [DEBUG] JournalWriteTask - follower789: journal writer stopped 09:27:28,846 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - follower789: journal closed 09:27:28,846 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRaftStorage - follower789: journal closed 09:27:28,846 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower789: stopped executor with 0 remaining tasks 09:27:28,846 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - follower789: stopped executor with 0 remaining tasks 09:27:28,846 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader787-collector792#-1243283292] 09:27:28,846 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader787] 09:27:28,846 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PekkoRaftStorage - leader787: terminating thread VirtualThread[#832,leader787-writer-162]/waiting 09:27:28,846 AM [leader787-writer-162] [DEBUG] JournalWriteTask - leader787: received 1 commands after 5.053 s 09:27:28,846 AM [leader787-writer-162] [DEBUG] JournalWriteTask - leader787: completed 1 commands in 16.64 μs 09:27:28,846 AM [leader787-writer-162] [DEBUG] JournalWriteTask - leader787: journal writer stopped 09:27:28,846 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] EntryJournalV1 - leader787: journal closed 09:27:28,846 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRaftStorage - leader787: journal closed 09:27:28,846 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: stopped executor with 0 remaining tasks 09:27:28,846 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftStorage - leader787: stopped executor with 0 remaining tasks [INFO] [07/05/2025 09:27:28.848] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 09:27:28,849 AM [test-pekko.actor.default-dispatcher-15] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [07/05/2025 09:27:28.849] [test-pekko.actor.default-dispatcher-15] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.171.247:17355) [INFO] [07/05/2025 09:27:28.851] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Exiting completed [INFO] [07/05/2025 09:27:28.851] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Shutting down... 09:27:28,851 AM [test-pekko.actor.default-dispatcher-15] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Exiting completed 09:27:28,851 AM [test-pekko.actor.default-dispatcher-15] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Shutting down... [INFO] [07/05/2025 09:27:28.851] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Successfully shut down 09:27:28,851 AM [test-pekko.actor.default-dispatcher-15] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Successfully shut down 09:27:28,852 AM [test-pekko.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:27:28,852 AM [test-pekko.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [07/05/2025 09:27:28.852] [test-pekko.remote.default-remote-dispatcher-10] [pekko://test@10.30.171.247:17355/system/remoting-terminator] Shutting down remote daemon. [INFO] [07/05/2025 09:27:28.852] [test-pekko.remote.default-remote-dispatcher-10] [pekko://test@10.30.171.247:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 09:27:28,855 AM [test-pekko.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [INFO] [07/05/2025 09:27:28.855] [test-pekko.remote.default-remote-dispatcher-10] [pekko://test@10.30.171.247:17355/system/remoting-terminator] Remoting shut down. [DEBUG] [07/05/2025 09:27:28.855] [test-pekko.actor.internal-dispatcher-3] [EventStream] shutting down: StandardOutLogger