[DEBUG] [06/27/2025 14:31:48.375] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 02:31:48,375 PM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [06/27/2025 14:31:48.375] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [06/27/2025 14:31:48.376] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [06/27/2025 14:31:48.390] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.8:17355] with UID [-5761347356960026607] 02:31:48,390 PM [test-pekko.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.8:17355] with UID [-5761347356960026607] [INFO] [06/27/2025 14:31:48.390] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.8:17355] - Starting up, Pekko version [1.0.3] ... 02:31:48,391 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.8:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [06/27/2025 14:31:48.393] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.8:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [06/27/2025 14:31:48.393] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.8:17355] - Started up successfully 02:31:48,394 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.8:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 02:31:48,394 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.8:17355] - Started up successfully [INFO] [06/27/2025 14:31:48.395] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.8:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [06/27/2025 14:31:48.395] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.8:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 02:31:48,395 PM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.8:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 02:31:48,395 PM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.8:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 02:31:48,406 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 02:31:48,406 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:31:48,412 PM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower279] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [06/27/2025 14:31:48.412] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower279]: org.opendaylight.controller.cluster.common.actor.Monitor@31f82a85 [INFO] [pekkoDeadLetter][06/27/2025 14:31:48.411] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower279] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,414 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower279: Starting recovery with journal batch size 1000 02:31:48,414 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: started executor 02:31:48,414 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: started executor 02:31:48,414 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower279: metadata journal open: replayFrom=1 applyTo=0 02:31:48,415 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower279: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:48,415 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower279: journal open: applyTo=0 02:31:48,415 PM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@543512d8 started 02:31:48,415 PM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: waiting for more work 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower279/TermInfo.properties does not exist 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower279/journal-v1-1.log 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower279/metadata-v1-1.log 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: no eligible files found 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower279: SnapshotOffer called. 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower279: initializing from snapshot taken at 2025-06-27T14:31:48.406Z 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=1, votedFor=null 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PeerInfos - follower279: Updated server config: isVoting: true, peers: [PeerInfo [id=follower281, address=pekko://test/user/follower281, votingState=NON_VOTING], PeerInfo [id=leader278, address=pekko://test/user/leader278, votingState=VOTING], PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=VOTING]] 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - Recovery snapshot applied for follower279 in 77.29 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower279: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:48.406Z 02:31:48,416 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower279: support() called 02:31:48,417 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower279/snapshot-00000000685EAB54-18331180.tmp [WARN] [06/27/2025 14:31:48.417] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower280]: org.opendaylight.controller.cluster.common.actor.Monitor@4d4c4a5d [INFO] [pekkoDeadLetter][06/27/2025 14:31:48.417] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower280] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,417 PM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower280] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,420 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower279/snapshot-00000000685EAB54-18331180.v1 02:31:48,420 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower279/snapshot-00000000685EAB54-18331180.v1 02:31:48,420 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower279/journal-v1-1.log 02:31:48,420 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower279/metadata-v1-1.log 02:31:48,420 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:48.406Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower279/snapshot-00000000685EAB54-18331180.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:48,420 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower279: local snapshot saved in 3.536 ms, deleting Pekko-persisted snapshots 02:31:48,420 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower280: Starting recovery with journal batch size 1000 02:31:48,420 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: started executor 02:31:48,420 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: started executor 02:31:48,420 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower279: Recovery completed in 4.302 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:48,421 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower280: metadata journal open: replayFrom=1 applyTo=0 02:31:48,421 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower280: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:48,421 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - follower280: journal open: applyTo=0 02:31:48,421 PM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@79da434c started 02:31:48,421 PM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: waiting for more work 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower280/TermInfo.properties does not exist 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower280/journal-v1-1.log 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower280/metadata-v1-1.log 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: no eligible files found 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower280: SnapshotOffer called. 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower280: initializing from snapshot taken at 2025-06-27T14:31:48.406Z 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=1, votedFor=null 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower280: Updated server config: isVoting: true, peers: [PeerInfo [id=follower281, address=pekko://test/user/follower281, votingState=NON_VOTING], PeerInfo [id=leader278, address=pekko://test/user/leader278, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for follower280 in 50.56 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower280: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:48.406Z 02:31:48,422 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower280: support() called [WARN] [06/27/2025 14:31:48.423] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower281]: org.opendaylight.controller.cluster.common.actor.Monitor@45c45391 02:31:48,423 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower280/snapshot-00000000685EAB54-18331180.tmp [INFO] [pekkoDeadLetter][06/27/2025 14:31:48.423] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower281] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,423 PM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower281] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,425 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=1, votedFor=null 02:31:48,425 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower279: Local TermInfo store seeded with TermInfo{term=1} 02:31:48,425 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower279: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@208071fd 02:31:48,425 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower279: Moving last applied index from -1 to -1 02:31:48,425 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower279: journal recovery completedin 320.5 μs with journalIndex=1 02:31:48,426 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower279: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:48,426 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower280/snapshot-00000000685EAB54-18331180.v1 02:31:48,427 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower280/snapshot-00000000685EAB54-18331180.v1 02:31:48,427 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower280/journal-v1-1.log 02:31:48,427 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower280/metadata-v1-1.log 02:31:48,427 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower280: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:48.406Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower280/snapshot-00000000685EAB54-18331180.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:48,427 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower280: local snapshot saved in 4.971 ms, deleting Pekko-persisted snapshots 02:31:48,428 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower280: Recovery completed in 6.124 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:48,431 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower281: Starting recovery with journal batch size 1000 02:31:48,431 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: started executor 02:31:48,431 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: started executor 02:31:48,431 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower281: metadata journal open: replayFrom=1 applyTo=0 02:31:48,432 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower281: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:48,432 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower281: journal open: applyTo=0 02:31:48,432 PM [follower281-writer-134] [DEBUG] JournalWriteTask - follower281: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3f13c7ca started 02:31:48,432 PM [follower281-writer-134] [DEBUG] JournalWriteTask - follower281: waiting for more work 02:31:48,432 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower281/TermInfo.properties does not exist 02:31:48,432 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower281/journal-v1-1.log 02:31:48,432 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower281/metadata-v1-1.log 02:31:48,432 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: no eligible files found 02:31:48,432 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower281: SnapshotOffer called. 02:31:48,432 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower281: initializing from snapshot taken at 2025-06-27T14:31:48.406Z 02:31:48,432 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower281: Set currentTerm=1, votedFor=null 02:31:48,432 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PeerInfos - follower281: Updated server config: isVoting: false, peers: [PeerInfo [id=leader278, address=pekko://test/user/leader278, votingState=VOTING], PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 02:31:48,433 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - Recovery snapshot applied for follower281 in 67.06 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:48,433 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower281: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:48.406Z 02:31:48,433 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower281: support() called 02:31:48,433 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower281/snapshot-00000000685EAB54-18331180.tmp [WARN] [06/27/2025 14:31:48.433] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader278]: org.opendaylight.controller.cluster.common.actor.Monitor@3598e5ae 02:31:48,433 PM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader278] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/27/2025 14:31:48.433] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader278] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,434 PM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - leader278: Starting recovery with journal batch size 1000 02:31:48,434 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: started executor 02:31:48,434 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=1, votedFor=null 02:31:48,434 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: started executor 02:31:48,434 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower280: Local TermInfo store seeded with TermInfo{term=1} 02:31:48,434 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower280: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@72540805 02:31:48,434 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower280: Moving last applied index from -1 to -1 02:31:48,435 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader278: metadata journal open: replayFrom=1 applyTo=0 02:31:48,435 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader278: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:48,435 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader278: journal open: applyTo=0 02:31:48,435 PM [leader278-writer-135] [DEBUG] JournalWriteTask - leader278: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@1d4dc05 started 02:31:48,435 PM [leader278-writer-135] [DEBUG] JournalWriteTask - leader278: waiting for more work 02:31:48,436 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower281/snapshot-00000000685EAB54-18331180.v1 02:31:48,436 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower281/snapshot-00000000685EAB54-18331180.v1 02:31:48,436 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower281/journal-v1-1.log 02:31:48,436 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower281/metadata-v1-1.log 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/leader278/TermInfo.properties does not exist 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/leader278/journal-v1-1.log 02:31:48,436 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:48.406Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/follower281/snapshot-00000000685EAB54-18331180.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/leader278/metadata-v1-1.log 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: no eligible files found 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - leader278: SnapshotOffer called. 02:31:48,436 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower281: local snapshot saved in 3.350 ms, deleting Pekko-persisted snapshots 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - leader278: initializing from snapshot taken at 2025-06-27T14:31:48.406Z 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader278: Set currentTerm=1, votedFor=null 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - leader278: Updated server config: isVoting: true, peers: [PeerInfo [id=follower281, address=pekko://test/user/follower281, votingState=NON_VOTING], PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for leader278 in 59.36 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader278: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:48.406Z 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader278: support() called 02:31:48,436 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/leader278/snapshot-00000000685EAB54-18331180.tmp 02:31:48,437 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower281: Recovery completed in 4.513 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:48,437 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower280: journal recovery completedin 2.552 ms with journalIndex=1 02:31:48,438 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower280: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:48,443 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/leader278/snapshot-00000000685EAB54-18331180.v1 02:31:48,443 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/leader278/snapshot-00000000685EAB54-18331180.v1 02:31:48,443 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/leader278/journal-v1-1.log 02:31:48,443 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/leader278/metadata-v1-1.log 02:31:48,443 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:48.406Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6571248946244046360/leader278/snapshot-00000000685EAB54-18331180.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:48,443 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader278: local snapshot saved in 6.922 ms, deleting Pekko-persisted snapshots 02:31:48,444 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader278: Recovery completed in 7.642 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:48,444 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower281: Set currentTerm=1, votedFor=null 02:31:48,444 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower281: Local TermInfo store seeded with TermInfo{term=1} 02:31:48,444 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower281: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6d857f35 02:31:48,444 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower281: Moving last applied index from -1 to -1 02:31:48,445 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower281: journal recovery completedin 218.1 μs with journalIndex=1 02:31:48,445 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower281: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:48,448 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader278: Set currentTerm=1, votedFor=null 02:31:48,448 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader278: Local TermInfo store seeded with TermInfo{term=1} 02:31:48,448 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader278: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@453af456 02:31:48,448 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader278: Moving last applied index from -1 to -1 02:31:48,449 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader278: journal recovery completedin 442.9 μs with journalIndex=1 02:31:48,449 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader278: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:48,869 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:31:48,869 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader278 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.8:17355, Removed) 02:31:48,869 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader278 (Follower): Received ElectionTimeout - switching to Candidate 02:31:48,869 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader278 (Candidate): Election: Candidate has following voting peers: [follower280, follower279] 02:31:48,873 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader278: Set currentTerm=2, votedFor=leader278 02:31:48,873 PM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - leader278 (Candidate): Starting new election term 2 02:31:48,873 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader278 (Candidate): Sending RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} to peer follower280 02:31:48,873 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader278 (Candidate): Sending RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} to peer follower279 02:31:48,873 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader278 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:31:48,873 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower280 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:48,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower279 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:48,873 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower280 (Follower): Cluster state: [] 02:31:48,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower279 (Follower): Cluster state: [] 02:31:48,873 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower280 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:48,873 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower279 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:48,873 PM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower279 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:48,873 PM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower280 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:48,876 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=2, votedFor=null 02:31:48,876 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower279 (Follower): In requestVote: RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:48,878 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=2, votedFor=null 02:31:48,878 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower280 (Follower): In requestVote: RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:48,879 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=2, votedFor=leader278 02:31:48,879 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower279 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:31:48,880 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader278 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:31:48,880 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader278 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:31:48,880 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Election: Leader has following peers: [follower281, follower280, follower279] 02:31:48,880 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower281: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:48,880 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower280: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:48,880 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower281 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader278, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:31:48,880 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower279: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:48,880 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader278 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:31:48,880 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower279: Last sync leader does not match current leader leader278, need to catch up to -1 [WARN] [06/27/2025 14:31:48.880] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower279] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower279, initialSyncDone=false] 02:31:48,881 PM [test-pekko.actor.default-dispatcher-17] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/27/2025 14:31:48.881] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower279] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,881 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower279 updated: matchIndex: -1, nextIndex: 0 02:31:48,881 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower279: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:48,881 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=2, votedFor=leader278 02:31:48,881 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower280 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:31:48,881 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower280: Last sync leader does not match current leader leader278, need to catch up to -1 02:31:48,881 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower280 updated: matchIndex: -1, nextIndex: 0 02:31:48,881 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower280: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:48.881] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower280] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower280, initialSyncDone=false] [INFO] [pekkoDeadLetter][06/27/2025 14:31:48.882] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower280] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,882 PM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,883 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower281: Set currentTerm=2, votedFor=null 02:31:48,883 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower281: Last sync leader does not match current leader leader278, need to catch up to -1 02:31:48,883 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader278 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower281 updated: matchIndex: -1, nextIndex: 0 [WARN] [06/27/2025 14:31:48.883] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower281] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower281, initialSyncDone=false] 02:31:48,883 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower281: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [INFO] [pekkoDeadLetter][06/27/2025 14:31:48.883] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower281] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,883 PM [test-pekko.actor.default-dispatcher-12] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,903 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:31:48,903 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader278 starting 02:31:48,906 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader278: onShutDown 02:31:48,907 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader278: Initiating leader transfer 02:31:48,907 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower281: onLeaderTransitioning: LeaderTransitioning [leaderId=leader278] 02:31:48,907 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower280: onLeaderTransitioning: LeaderTransitioning [leaderId=leader278] 02:31:48,907 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower279: onLeaderTransitioning: LeaderTransitioning [leaderId=leader278] 02:31:48,907 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader278: pauseLeader successfully completed - doing transfer 02:31:48,907 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Leader - leader278 (Leader): Attempting to transfer leadership 02:31:48,907 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower281: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:48,907 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower280: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:48,907 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 02:31:48,907 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower279: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:48,907 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) [WARN] [06/27/2025 14:31:48.907] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower281] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower281, initialSyncDone=true] [INFO] [pekkoDeadLetter][06/27/2025 14:31:48.907] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower281] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,907 PM [test-pekko.actor.default-dispatcher-16] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,907 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) [WARN] [06/27/2025 14:31:48.908] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower279] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower279, initialSyncDone=true] [WARN] [06/27/2025 14:31:48.908] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower280] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower280, initialSyncDone=true] [INFO] [pekkoDeadLetter][06/27/2025 14:31:48.908] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower279] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,908 PM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/27/2025 14:31:48.908] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower280] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:48,908 PM [test-pekko.actor.default-dispatcher-15] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:31:49,0 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 02:31:49,0 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 02:31:49,0 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 02:31:49,119 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 02:31:49,119 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 02:31:49,120 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 02:31:49,240 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader278: leader transfer aborted 02:31:49,240 PM [test-pekko.actor.default-dispatcher-16] [WARN] RaftActorLeadershipTransferCohort - leader278: Failed to transfer leadership in 333.0 ms 02:31:49,240 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader278: leader transfer failed - sending PoisonPill 02:31:49,240 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Leader - leader278 (Leader): Leadership transfer expired 02:31:49,240 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - leader278: terminating thread VirtualThread[#706,leader278-writer-135]/waiting 02:31:49,240 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 02:31:49,240 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 02:31:49,240 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 02:31:49,240 PM [leader278-writer-135] [DEBUG] JournalWriteTask - leader278: received 1 entries in 805.0 ms 02:31:49,240 PM [leader278-writer-135] [DEBUG] JournalWriteTask - leader278: persisted 1 entries in 46.01 μs 02:31:49,241 PM [leader278-writer-135] [DEBUG] JournalWriteTask - leader278: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@1d4dc05 stopped 02:31:49,241 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader278: journal closed 02:31:49,241 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - leader278: journal closed 02:31:49,241 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: stopped executor with 0 remaining tasks 02:31:49,241 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: stopped executor with 0 remaining tasks [WARN] [06/27/2025 14:31:49.241] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/leader278] received dead letter from TestActor[pekko://test/user/follower280]: AppendEntriesReply{term=2, success=true, followerId=follower280, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} [WARN] [06/27/2025 14:31:49.241] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/leader278] received dead letter from TestActor[pekko://test/user/follower281]: AppendEntriesReply{term=2, success=true, followerId=follower281, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} [WARN] [06/27/2025 14:31:49.241] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/leader278] received dead letter from TestActor[pekko://test/user/follower279]: AppendEntriesReply{term=2, success=true, followerId=follower279, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:31:49,242 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader278 ending 02:31:49,245 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 02:31:49,245 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower279-notifier282#1618731652] 02:31:49,257 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower279-collector283#1684125601] 02:31:49,257 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower279] 02:31:49,257 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower279: terminating thread VirtualThread[#700,follower279-writer-132]/waiting 02:31:49,257 PM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: received 1 entries in 842.2 ms 02:31:49,257 PM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: persisted 1 entries in 40.94 μs 02:31:49,258 PM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@543512d8 stopped 02:31:49,258 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower279: journal closed 02:31:49,258 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower279: journal closed 02:31:49,258 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower279: stopped executor with 0 remaining tasks 02:31:49,258 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower279: stopped executor with 0 remaining tasks 02:31:49,258 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower280-notifier284#-1311170531] 02:31:49,258 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower280-collector285#-919246118] 02:31:49,259 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower280] 02:31:49,259 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower280: terminating thread VirtualThread[#701,follower280-writer-133]/waiting 02:31:49,259 PM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: received 1 entries in 837.9 ms 02:31:49,259 PM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: persisted 1 entries in 19.33 μs 02:31:49,259 PM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@79da434c stopped 02:31:49,259 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower280: journal closed 02:31:49,259 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower280: journal closed 02:31:49,259 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: stopped executor with 0 remaining tasks 02:31:49,259 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: stopped executor with 0 remaining tasks 02:31:49,260 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower281-notifier286#-708628920] 02:31:49,260 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower281-collector287#378809074] 02:31:49,260 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower281] 02:31:49,260 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower281: terminating thread VirtualThread[#705,follower281-writer-134]/waiting 02:31:49,260 PM [follower281-writer-134] [DEBUG] JournalWriteTask - follower281: received 1 entries in 828.5 ms 02:31:49,261 PM [follower281-writer-134] [DEBUG] JournalWriteTask - follower281: persisted 1 entries in 20.70 μs 02:31:49,261 PM [follower281-writer-134] [DEBUG] JournalWriteTask - follower281: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3f13c7ca stopped 02:31:49,261 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower281: journal closed 02:31:49,261 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower281: journal closed 02:31:49,261 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower281: stopped executor with 0 remaining tasks 02:31:49,261 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower281: stopped executor with 0 remaining tasks 02:31:49,261 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader278-notifier288#-1318894489] 02:31:49,261 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader278-collector289#-1910855214] 02:31:49,262 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader278] 02:31:49,265 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 02:31:49,265 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/27/2025 14:31:49.268] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower291]: org.opendaylight.controller.cluster.common.actor.Monitor@3701792c 02:31:49,268 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower291: Starting recovery with journal batch size 1000 02:31:49,268 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: started executor 02:31:49,269 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: started executor 02:31:49,269 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower291: metadata journal open: replayFrom=1 applyTo=0 02:31:49,270 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower291: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:49,270 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower291: journal open: applyTo=0 02:31:49,270 PM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@7d0bad35 started 02:31:49,270 PM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: waiting for more work 02:31:49,270 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower291/TermInfo.properties does not exist 02:31:49,270 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower291/journal-v1-1.log 02:31:49,270 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower291/metadata-v1-1.log 02:31:49,270 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: no eligible files found 02:31:49,270 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower291: SnapshotOffer called. 02:31:49,271 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower291: initializing from snapshot taken at 2025-06-27T14:31:49.265Z 02:31:49,271 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=1, votedFor=null 02:31:49,271 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower291: Updated server config: isVoting: true, peers: [PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=VOTING], PeerInfo [id=follower293, address=pekko://test/user/follower293, votingState=NON_VOTING], PeerInfo [id=leader290, address=pekko://test/user/leader290, votingState=VOTING]] 02:31:49,271 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - Recovery snapshot applied for follower291 in 80.50 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:49,271 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower291: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:49.265Z 02:31:49,271 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower291: support() called 02:31:49,271 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower291/snapshot-00000000685EAB55-0FCB9440.tmp [WARN] [06/27/2025 14:31:49.272] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower292]: org.opendaylight.controller.cluster.common.actor.Monitor@1ec3219e 02:31:49,273 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower292: Starting recovery with journal batch size 1000 02:31:49,273 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: started executor 02:31:49,273 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: started executor 02:31:49,274 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower292: metadata journal open: replayFrom=1 applyTo=0 02:31:49,274 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower291/snapshot-00000000685EAB55-0FCB9440.v1 02:31:49,275 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower291/snapshot-00000000685EAB55-0FCB9440.v1 02:31:49,275 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower292: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:49,275 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower291/journal-v1-1.log 02:31:49,275 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower292: journal open: applyTo=0 02:31:49,275 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower291/metadata-v1-1.log 02:31:49,275 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower291: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:49.265Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower291/snapshot-00000000685EAB55-0FCB9440.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:49,275 PM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@56e041fb started 02:31:49,275 PM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: waiting for more work 02:31:49,275 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower291: local snapshot saved in 3.930 ms, deleting Pekko-persisted snapshots 02:31:49,276 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower291: Recovery completed in 5.135 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower292/TermInfo.properties does not exist 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower292/journal-v1-1.log 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower292/metadata-v1-1.log 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: no eligible files found 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower292: SnapshotOffer called. 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower292: initializing from snapshot taken at 2025-06-27T14:31:49.265Z 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=1, votedFor=null 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - follower292: Updated server config: isVoting: true, peers: [PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=follower293, address=pekko://test/user/follower293, votingState=NON_VOTING], PeerInfo [id=leader290, address=pekko://test/user/leader290, votingState=VOTING]] 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for follower292 in 96.88 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower292: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:49.265Z 02:31:49,276 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower292: support() called 02:31:49,277 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower292/snapshot-00000000685EAB55-0FCB9440.tmp [WARN] [06/27/2025 14:31:49.277] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower293]: org.opendaylight.controller.cluster.common.actor.Monitor@6715136d 02:31:49,278 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower293: Starting recovery with journal batch size 1000 02:31:49,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=1, votedFor=null 02:31:49,279 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower291: Local TermInfo store seeded with TermInfo{term=1} 02:31:49,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower291: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@43f03518 02:31:49,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower291: Moving last applied index from -1 to -1 02:31:49,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower291: journal recovery completedin 239.1 μs with journalIndex=1 02:31:49,279 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower291: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:49,279 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: started executor 02:31:49,280 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: started executor 02:31:49,280 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower293: metadata journal open: replayFrom=1 applyTo=0 02:31:49,280 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower292/snapshot-00000000685EAB55-0FCB9440.v1 02:31:49,280 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower292/snapshot-00000000685EAB55-0FCB9440.v1 02:31:49,280 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower292/journal-v1-1.log 02:31:49,280 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower292/metadata-v1-1.log 02:31:49,280 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:49.265Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower292/snapshot-00000000685EAB55-0FCB9440.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:49,280 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower292: local snapshot saved in 4.078 ms, deleting Pekko-persisted snapshots 02:31:49,281 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower292: Recovery completed in 4.928 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:49,282 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower293: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:49,282 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower293: journal open: applyTo=0 02:31:49,282 PM [follower293-writer-138] [DEBUG] JournalWriteTask - follower293: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@776cf17a started 02:31:49,282 PM [follower293-writer-138] [DEBUG] JournalWriteTask - follower293: waiting for more work 02:31:49,282 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower293/TermInfo.properties does not exist 02:31:49,282 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower293/journal-v1-1.log 02:31:49,282 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower293/metadata-v1-1.log 02:31:49,282 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: no eligible files found 02:31:49,282 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower293: SnapshotOffer called. 02:31:49,282 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower293: initializing from snapshot taken at 2025-06-27T14:31:49.265Z 02:31:49,283 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=1, votedFor=null 02:31:49,283 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower293: Updated server config: isVoting: false, peers: [PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=VOTING], PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=leader290, address=pekko://test/user/leader290, votingState=VOTING]] 02:31:49,283 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=1, votedFor=null 02:31:49,283 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower292: Local TermInfo store seeded with TermInfo{term=1} 02:31:49,283 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower292: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@48c3f841 02:31:49,283 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower292: Moving last applied index from -1 to -1 [WARN] [06/27/2025 14:31:49.283] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader290]: org.opendaylight.controller.cluster.common.actor.Monitor@67b6cc1d 02:31:49,283 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower292: journal recovery completedin 249.4 μs with journalIndex=1 02:31:49,284 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower292: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:49,284 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - leader290: Starting recovery with journal batch size 1000 02:31:49,284 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: started executor 02:31:49,284 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: started executor 02:31:49,284 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader290: metadata journal open: replayFrom=1 applyTo=0 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader290: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader290: journal open: applyTo=0 02:31:49,285 PM [leader290-writer-139] [DEBUG] JournalWriteTask - leader290: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@50ff2f66 started 02:31:49,285 PM [leader290-writer-139] [DEBUG] JournalWriteTask - leader290: waiting for more work 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/leader290/TermInfo.properties does not exist 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/leader290/journal-v1-1.log 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/leader290/metadata-v1-1.log 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: no eligible files found 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - leader290: SnapshotOffer called. 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - leader290: initializing from snapshot taken at 2025-06-27T14:31:49.265Z 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=1, votedFor=null 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PeerInfos - leader290: Updated server config: isVoting: true, peers: [PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=VOTING], PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=follower293, address=pekko://test/user/follower293, votingState=NON_VOTING]] 02:31:49,285 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - Recovery snapshot applied for leader290 in 67.89 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:49,285 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - Recovery snapshot applied for follower293 in 2.871 ms: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:49,286 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader290: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:49.265Z 02:31:49,286 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower293: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:49.265Z 02:31:49,286 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader290: support() called 02:31:49,286 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower293: support() called 02:31:49,286 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/leader290/snapshot-00000000685EAB55-0FCB9440.tmp 02:31:49,286 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower293/snapshot-00000000685EAB55-0FCB9440.tmp 02:31:49,289 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower293/snapshot-00000000685EAB55-0FCB9440.v1 02:31:49,289 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower293/snapshot-00000000685EAB55-0FCB9440.v1 02:31:49,289 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower293/journal-v1-1.log 02:31:49,289 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower293/metadata-v1-1.log 02:31:49,289 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower293: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:49.265Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/follower293/snapshot-00000000685EAB55-0FCB9440.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:49,289 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower293: local snapshot saved in 3.721 ms, deleting Pekko-persisted snapshots 02:31:49,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/leader290/snapshot-00000000685EAB55-0FCB9440.v1 02:31:49,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/leader290/snapshot-00000000685EAB55-0FCB9440.v1 02:31:49,290 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower293: Recovery completed in 7.326 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:49,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/leader290/journal-v1-1.log 02:31:49,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/leader290/metadata-v1-1.log 02:31:49,290 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader290: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:49.265Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest14315823724541546669/leader290/snapshot-00000000685EAB55-0FCB9440.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:49,290 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader290: local snapshot saved in 4.468 ms, deleting Pekko-persisted snapshots 02:31:49,291 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - leader290: Recovery completed in 5.178 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:49,293 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=1, votedFor=null 02:31:49,293 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower293: Local TermInfo store seeded with TermInfo{term=1} 02:31:49,293 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower293: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@24f8fedc 02:31:49,293 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower293: Moving last applied index from -1 to -1 02:31:49,293 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower293: journal recovery completedin 274.1 μs with journalIndex=1 02:31:49,294 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=1, votedFor=null 02:31:49,294 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - leader290: Local TermInfo store seeded with TermInfo{term=1} 02:31:49,294 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader290: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@5e7e12f9 02:31:49,294 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader290: Moving last applied index from -1 to -1 02:31:49,294 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower293: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:49,294 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - leader290: journal recovery completedin 232.7 μs with journalIndex=1 02:31:49,295 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader290: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:49,689 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:31:49,689 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader290 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.8:17355, Removed) 02:31:49,689 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - leader290 (Follower): Received ElectionTimeout - switching to Candidate 02:31:49,690 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader290 (Candidate): Election: Candidate has following voting peers: [follower292, follower291] 02:31:49,693 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=2, votedFor=leader290 02:31:49,693 PM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - leader290 (Candidate): Starting new election term 2 02:31:49,693 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader290 (Candidate): Sending RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} to peer follower292 02:31:49,693 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader290 (Candidate): Sending RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} to peer follower291 02:31:49,694 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower292 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:49,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower291 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:49,694 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader290 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:31:49,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower291 (Follower): Cluster state: [] 02:31:49,694 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower292 (Follower): Cluster state: [] 02:31:49,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower291 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:49,694 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower292 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:49,694 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower291 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:49,694 PM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower292 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:49,697 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=2, votedFor=null 02:31:49,697 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower291 (Follower): In requestVote: RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:49,698 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=2, votedFor=null 02:31:49,698 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower292 (Follower): In requestVote: RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:49,700 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=2, votedFor=leader290 02:31:49,700 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower291 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:31:49,700 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader290 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:31:49,700 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader290 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:31:49,701 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Election: Leader has following peers: [follower292, follower291, follower293] 02:31:49,701 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,701 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,701 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,701 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower291: Last sync leader does not match current leader leader290, need to catch up to -1 02:31:49,701 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - leader290 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:31:49,701 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=2, votedFor=leader290 02:31:49,701 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower293 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader290, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:31:49,701 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower292 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} [WARN] [06/27/2025 14:31:49.701] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=false] 02:31:49,701 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower292: Last sync leader does not match current leader leader290, need to catch up to -1 [WARN] [06/27/2025 14:31:49.701] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower292] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower292, initialSyncDone=false] 02:31:49,702 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower291 updated: matchIndex: -1, nextIndex: 0 02:31:49,702 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,702 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower292 updated: matchIndex: -1, nextIndex: 0 02:31:49,702 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,704 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=2, votedFor=null 02:31:49,704 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower293: Last sync leader does not match current leader leader290, need to catch up to -1 02:31:49,704 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader290 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower293 updated: matchIndex: -1, nextIndex: 0 [WARN] [06/27/2025 14:31:49.704] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower293] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower293, initialSyncDone=false] 02:31:49,704 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,748 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:31:49,748 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 02:31:49,749 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 02:31:49,749 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 02:31:49,749 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 02:31:49,749 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader290: onRequestLeadership RequestLeadership [requestedFollowerId=follower292, replyTo=Actor[pekko://test/user/$a#525467222]] 02:31:49,751 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader290: Initiating leader transfer 02:31:49,751 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower292: onLeaderTransitioning: LeaderTransitioning [leaderId=leader290] 02:31:49,751 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower291: onLeaderTransitioning: LeaderTransitioning [leaderId=leader290] 02:31:49,751 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader290: pauseLeader successfully completed - doing transfer 02:31:49,751 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower293: onLeaderTransitioning: LeaderTransitioning [leaderId=leader290] 02:31:49,751 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Leader - leader290 (Leader): Attempting to transfer leadership 02:31:49,751 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,751 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,752 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,752 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower292: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 02:31:49,752 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower291: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 02:31:49,752 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower293: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) [WARN] [06/27/2025 14:31:49.752] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower292] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower292, initialSyncDone=true] [WARN] [06/27/2025 14:31:49.752] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=true] [WARN] [06/27/2025 14:31:49.752] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower293] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower293, initialSyncDone=true] 02:31:49,752 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Leader - leader290 (Leader): tryToCompleteLeadershipTransfer: followerId: follower292, matchIndex: -1, lastIndex: -1, isVoting: true 02:31:49,752 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Leader - leader290 (Leader): Follower's log matches - sending ElectionTimeout 02:31:49,752 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,752 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,752 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,752 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower291: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 02:31:49,752 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Leader - leader290 (Leader): Leader transfer complete 02:31:49,752 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower293: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 02:31:49,752 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader290: leader transfer complete - waiting for new leader 02:31:49,753 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower292: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 02:31:49,753 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:31:49,753 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - follower292 (Follower): Received TimeoutNow - switching to Candidate 02:31:49,753 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - follower292 (Candidate): Election: Candidate has following voting peers: [follower291, leader290] 02:31:49,756 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=3, votedFor=follower292 02:31:49,756 PM [test-pekko.actor.default-dispatcher-18] [INFO] Candidate - follower292 (Candidate): Starting new election term 3 02:31:49,756 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - follower292 (Candidate): Sending RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} to peer follower291 02:31:49,756 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - follower292 (Candidate): Sending RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} to peer leader290 02:31:49,756 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower291 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:49,756 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - follower292 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 02:31:49,756 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower291 (Follower): Cluster state: [] 02:31:49,756 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower291 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:49,756 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower291 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 2 - updating term 02:31:49,756 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader290 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:49,756 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader290 (Leader): Cluster state: [] 02:31:49,756 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader290 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:49,756 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractLeader - leader290 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower 02:31:49,759 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=3, votedFor=null 02:31:49,759 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower291 (Follower): In requestVote: RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:49,760 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=3, votedFor=null 02:31:49,760 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader290 (Leader): Leadership transfer in progress - processing RequestVote 02:31:49,760 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader290 (Leader): In requestVote: RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:49,761 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=3, votedFor=follower292 02:31:49,761 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower291 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 02:31:49,761 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - follower292 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 02:31:49,761 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - follower292 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 02:31:49,761 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - follower292 (Leader): Election: Leader has following peers: [follower291, follower293, leader290] 02:31:49,762 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower291: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,762 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower293: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,762 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower leader290: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,762 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - follower292 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 02:31:49,762 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower293 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower292, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 02:31:49,762 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=3, votedFor=follower292 02:31:49,762 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower291: Last sync leader does not match current leader follower292, need to catch up to -1 02:31:49,762 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader290 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 02:31:49,762 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower292 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower291 updated: matchIndex: -1, nextIndex: 0 02:31:49,762 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader290 (Leader) :- Switching from behavior Leader to Follower, election term: 3 02:31:49,762 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:49.762] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=false] 02:31:49,762 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - leader290: Last sync leader does not match current leader follower292, need to catch up to -1 02:31:49,762 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower292 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader290 updated: matchIndex: -1, nextIndex: 0 02:31:49,762 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader290: leader changed to follower292 02:31:49,762 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower leader290: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:49,762 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorLeadershipTransferCohort - leader290: Successfully transferred leadership to follower292 in 11.39 ms 02:31:49,762 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader290: Leadership transferred successfully to follower292 [WARN] [06/27/2025 14:31:49.763] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/leader290] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader290, initialSyncDone=false] 02:31:49,764 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=3, votedFor=null 02:31:49,764 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower293: Last sync leader does not match current leader follower292, need to catch up to -1 02:31:49,764 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - follower292 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower293 updated: matchIndex: -1, nextIndex: 0 02:31:49,764 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:49.764] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower293] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower293, initialSyncDone=false] 02:31:49,801 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 02:31:49,802 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower291-notifier294#974706439] 02:31:49,813 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower291-collector295#-2068873587] 02:31:49,814 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower291] 02:31:49,814 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower291: terminating thread VirtualThread[#707,follower291-writer-136]/waiting 02:31:49,814 PM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: received 1 entries in 544.3 ms 02:31:49,814 PM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: persisted 1 entries in 49.16 μs 02:31:49,814 PM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@7d0bad35 stopped 02:31:49,815 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower291: journal closed 02:31:49,815 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower291: journal closed 02:31:49,815 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower291: stopped executor with 0 remaining tasks 02:31:49,815 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower291: stopped executor with 0 remaining tasks 02:31:49,815 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower292-notifier296#874198312] 02:31:49,815 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower292-collector297#1356693171] 02:31:49,816 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower292] 02:31:49,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower292: terminating thread VirtualThread[#708,follower292-writer-137]/waiting 02:31:49,816 PM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: received 1 entries in 541.1 ms 02:31:49,816 PM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: persisted 1 entries in 27.29 μs 02:31:49,816 PM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@56e041fb stopped 02:31:49,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower292: journal closed 02:31:49,816 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower292: journal closed 02:31:49,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower292: stopped executor with 0 remaining tasks 02:31:49,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower292: stopped executor with 0 remaining tasks 02:31:49,816 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower293-notifier298#-736489615] 02:31:49,817 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower293-collector299#215881655] 02:31:49,817 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower293] 02:31:49,817 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower293: terminating thread VirtualThread[#709,follower293-writer-138]/waiting 02:31:49,817 PM [follower293-writer-138] [DEBUG] JournalWriteTask - follower293: received 1 entries in 535.5 ms 02:31:49,817 PM [follower293-writer-138] [DEBUG] JournalWriteTask - follower293: persisted 1 entries in 27.56 μs 02:31:49,817 PM [follower293-writer-138] [DEBUG] JournalWriteTask - follower293: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@776cf17a stopped 02:31:49,817 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower293: journal closed 02:31:49,817 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower293: journal closed 02:31:49,818 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower293: stopped executor with 0 remaining tasks 02:31:49,818 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower293: stopped executor with 0 remaining tasks 02:31:49,818 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader290-notifier300#2037036719] 02:31:49,818 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader290-collector301#317116097] 02:31:49,818 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader290] 02:31:49,818 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - leader290: terminating thread VirtualThread[#710,leader290-writer-139]/waiting 02:31:49,818 PM [leader290-writer-139] [DEBUG] JournalWriteTask - leader290: received 1 entries in 533.5 ms 02:31:49,818 PM [leader290-writer-139] [DEBUG] JournalWriteTask - leader290: persisted 1 entries in 17.60 μs 02:31:49,818 PM [leader290-writer-139] [DEBUG] JournalWriteTask - leader290: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@50ff2f66 stopped 02:31:49,819 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader290: journal closed 02:31:49,819 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - leader290: journal closed 02:31:49,819 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader290: stopped executor with 0 remaining tasks 02:31:49,819 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader290: stopped executor with 0 remaining tasks 02:31:49,819 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$a#525467222] 02:31:49,821 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 02:31:49,821 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/27/2025 14:31:49.822] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower303]: org.opendaylight.controller.cluster.common.actor.Monitor@19874245 02:31:49,823 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower303: Starting recovery with journal batch size 1000 02:31:49,823 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: started executor 02:31:49,823 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: started executor 02:31:49,824 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower303: metadata journal open: replayFrom=1 applyTo=0 02:31:49,824 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower303: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:49,824 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower303: journal open: applyTo=0 02:31:49,824 PM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@70bd4249 started 02:31:49,824 PM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: waiting for more work 02:31:49,825 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower303/TermInfo.properties does not exist 02:31:49,825 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower303/journal-v1-1.log 02:31:49,825 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower303/metadata-v1-1.log 02:31:49,825 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: no eligible files found 02:31:49,825 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower303: SnapshotOffer called. 02:31:49,825 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower303: initializing from snapshot taken at 2025-06-27T14:31:49.821Z 02:31:49,825 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=1, votedFor=null 02:31:49,825 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - follower303: Updated server config: isVoting: true, peers: [PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=VOTING], PeerInfo [id=leader302, address=pekko://test/user/leader302, votingState=VOTING], PeerInfo [id=follower305, address=pekko://test/user/follower305, votingState=NON_VOTING]] 02:31:49,825 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - Recovery snapshot applied for follower303 in 110.6 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:49,826 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower303: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:49.821Z 02:31:49,826 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower303: support() called 02:31:49,826 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower303/snapshot-00000000685EAB55-30EF7740.tmp [WARN] [06/27/2025 14:31:49.826] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower304]: org.opendaylight.controller.cluster.common.actor.Monitor@4880b34c 02:31:49,827 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower304: Starting recovery with journal batch size 1000 02:31:49,828 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower304: started executor 02:31:49,828 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower304: started executor 02:31:49,828 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower304: metadata journal open: replayFrom=1 applyTo=0 02:31:49,828 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower303/snapshot-00000000685EAB55-30EF7740.v1 02:31:49,828 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower303/snapshot-00000000685EAB55-30EF7740.v1 02:31:49,829 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower303/journal-v1-1.log 02:31:49,829 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower303/metadata-v1-1.log 02:31:49,829 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:49.821Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower303/snapshot-00000000685EAB55-30EF7740.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:49,829 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower304: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:49,829 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower303: local snapshot saved in 3.137 ms, deleting Pekko-persisted snapshots 02:31:49,829 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower304: journal open: applyTo=0 02:31:49,829 PM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@24063a3c started 02:31:49,829 PM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: waiting for more work 02:31:49,829 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower304/TermInfo.properties does not exist 02:31:49,829 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower304/journal-v1-1.log 02:31:49,829 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower304/metadata-v1-1.log 02:31:49,830 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: no eligible files found 02:31:49,830 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower304: SnapshotOffer called. 02:31:49,830 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower304: initializing from snapshot taken at 2025-06-27T14:31:49.821Z 02:31:49,830 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=1, votedFor=null 02:31:49,830 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - follower304: Updated server config: isVoting: true, peers: [PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=leader302, address=pekko://test/user/leader302, votingState=VOTING], PeerInfo [id=follower305, address=pekko://test/user/follower305, votingState=NON_VOTING]] 02:31:49,830 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower303: Recovery completed in 4.485 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:49,830 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - Recovery snapshot applied for follower304 in 508.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:49,830 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower304: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:49.821Z 02:31:49,830 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower304: support() called 02:31:49,831 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower304/snapshot-00000000685EAB55-30EF7740.tmp 02:31:49,832 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=1, votedFor=null 02:31:49,832 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower303: Local TermInfo store seeded with TermInfo{term=1} 02:31:49,832 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower303: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@5cff3182 02:31:49,832 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower303: Moving last applied index from -1 to -1 02:31:49,832 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - follower303: journal recovery completedin 216.3 μs with journalIndex=1 02:31:49,833 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower303: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) [WARN] [06/27/2025 14:31:49.834] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower305]: org.opendaylight.controller.cluster.common.actor.Monitor@7133e604 02:31:49,834 PM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - follower305: Starting recovery with journal batch size 1000 02:31:49,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower304/snapshot-00000000685EAB55-30EF7740.v1 02:31:49,834 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: started executor 02:31:49,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower304/snapshot-00000000685EAB55-30EF7740.v1 02:31:49,834 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: started executor 02:31:49,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower304/journal-v1-1.log 02:31:49,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower304/metadata-v1-1.log 02:31:49,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:49.821Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower304/snapshot-00000000685EAB55-30EF7740.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:49,834 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower304: local snapshot saved in 4.172 ms, deleting Pekko-persisted snapshots 02:31:49,835 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower305: metadata journal open: replayFrom=1 applyTo=0 02:31:49,835 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower305: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:49,835 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower305: journal open: applyTo=0 02:31:49,835 PM [follower305-writer-142] [DEBUG] JournalWriteTask - follower305: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3471f8e7 started 02:31:49,835 PM [follower305-writer-142] [DEBUG] JournalWriteTask - follower305: waiting for more work 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower305/TermInfo.properties does not exist 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower305/journal-v1-1.log 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower305/metadata-v1-1.log 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: no eligible files found 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower305: SnapshotOffer called. 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower305: initializing from snapshot taken at 2025-06-27T14:31:49.821Z 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower305: Set currentTerm=1, votedFor=null 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower305: Updated server config: isVoting: false, peers: [PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=VOTING], PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=leader302, address=pekko://test/user/leader302, votingState=VOTING]] 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower305 in 70.38 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower305: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:49.821Z 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower305: support() called 02:31:49,836 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower305/snapshot-00000000685EAB55-30EF7740.tmp 02:31:49,837 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower304: Recovery completed in 7.339 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:49,838 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower305/snapshot-00000000685EAB55-30EF7740.v1 02:31:49,839 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower305/snapshot-00000000685EAB55-30EF7740.v1 02:31:49,839 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower305/journal-v1-1.log 02:31:49,839 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower305/metadata-v1-1.log 02:31:49,839 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower305: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:49.821Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/follower305/snapshot-00000000685EAB55-30EF7740.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:49,839 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower305: local snapshot saved in 2.818 ms, deleting Pekko-persisted snapshots 02:31:49,839 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=1, votedFor=null 02:31:49,839 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower304: Local TermInfo store seeded with TermInfo{term=1} 02:31:49,839 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower304: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@34dee1b8 02:31:49,839 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower304: Moving last applied index from -1 to -1 02:31:49,839 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower305: Recovery completed in 3.716 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [06/27/2025 14:31:49.840] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader302]: org.opendaylight.controller.cluster.common.actor.Monitor@7c011f29 02:31:49,840 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower304: journal recovery completedin 678.4 μs with journalIndex=1 02:31:49,840 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - leader302: Starting recovery with journal batch size 1000 02:31:49,840 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower304: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:49,840 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader302: started executor 02:31:49,841 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader302: started executor 02:31:49,841 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader302: metadata journal open: replayFrom=1 applyTo=0 02:31:49,841 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower305: Set currentTerm=1, votedFor=null 02:31:49,841 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower305: Local TermInfo store seeded with TermInfo{term=1} 02:31:49,841 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower305: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@43273abe 02:31:49,841 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower305: Moving last applied index from -1 to -1 02:31:49,841 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader302: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:49,841 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - leader302: journal open: applyTo=0 02:31:49,842 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower305: journal recovery completedin 196.8 μs with journalIndex=1 02:31:49,842 PM [leader302-writer-143] [DEBUG] JournalWriteTask - leader302: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@1d5c7510 started 02:31:49,842 PM [leader302-writer-143] [DEBUG] JournalWriteTask - leader302: waiting for more work 02:31:49,842 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower305: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:49,842 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/leader302/TermInfo.properties does not exist 02:31:49,842 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/leader302/journal-v1-1.log 02:31:49,842 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/leader302/metadata-v1-1.log 02:31:49,842 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: no eligible files found 02:31:49,842 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - leader302: SnapshotOffer called. 02:31:49,842 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - leader302: initializing from snapshot taken at 2025-06-27T14:31:49.821Z 02:31:49,842 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader302: Set currentTerm=1, votedFor=null 02:31:49,842 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - leader302: Updated server config: isVoting: true, peers: [PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=VOTING], PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=follower305, address=pekko://test/user/follower305, votingState=NON_VOTING]] 02:31:49,843 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - Recovery snapshot applied for leader302 in 108.3 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:49,843 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader302: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:49.821Z 02:31:49,843 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - leader302: support() called 02:31:49,843 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/leader302/snapshot-00000000685EAB55-30EF7740.tmp 02:31:49,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/leader302/snapshot-00000000685EAB55-30EF7740.v1 02:31:49,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/leader302/snapshot-00000000685EAB55-30EF7740.v1 02:31:49,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/leader302/journal-v1-1.log 02:31:49,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/leader302/metadata-v1-1.log 02:31:49,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:49.821Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest3051014609523288345/leader302/snapshot-00000000685EAB55-30EF7740.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:49,846 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader302: local snapshot saved in 3.059 ms, deleting Pekko-persisted snapshots 02:31:49,848 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader302: Recovery completed in 5.170 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:49,851 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader302: Set currentTerm=1, votedFor=null 02:31:49,851 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader302: Local TermInfo store seeded with TermInfo{term=1} 02:31:49,851 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader302: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@64f2a606 02:31:49,851 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader302: Moving last applied index from -1 to -1 02:31:49,851 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader302: journal recovery completedin 174.3 μs with journalIndex=1 02:31:49,852 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader302: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:50,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:31:50,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - leader302 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.8:17355, Removed) 02:31:50,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - leader302 (Follower): Received ElectionTimeout - switching to Candidate 02:31:50,260 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader302 (Candidate): Election: Candidate has following voting peers: [follower304, follower303] 02:31:50,264 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader302: Set currentTerm=2, votedFor=leader302 02:31:50,264 PM [test-pekko.actor.default-dispatcher-16] [INFO] Candidate - leader302 (Candidate): Starting new election term 2 02:31:50,264 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader302 (Candidate): Sending RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} to peer follower304 02:31:50,264 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader302 (Candidate): Sending RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} to peer follower303 02:31:50,264 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader302 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:31:50,264 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower304 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:50,264 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower304 (Follower): Cluster state: [] 02:31:50,264 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower303 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:50,264 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower304 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:50,264 PM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower304 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:50,264 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower303 (Follower): Cluster state: [] 02:31:50,264 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower303 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:50,264 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower303 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:50,268 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=2, votedFor=null 02:31:50,268 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower304 (Follower): In requestVote: RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:50,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=2, votedFor=leader302 02:31:50,271 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=2, votedFor=null 02:31:50,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower304 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:31:50,271 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower303 (Follower): In requestVote: RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:50,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader302 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:31:50,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader302 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:31:50,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader302 (Leader): Election: Leader has following peers: [follower304, follower303, follower305] 02:31:50,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower304: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:50,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower303: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:50,271 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower304: Last sync leader does not match current leader leader302, need to catch up to -1 02:31:50,271 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower305: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:50,271 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader302 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:31:50,272 PM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower305 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:31:50,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader302 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower304 updated: matchIndex: -1, nextIndex: 0 [WARN] [06/27/2025 14:31:50.272] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower304] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower304, initialSyncDone=false] 02:31:50,272 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower304: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:50,274 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=2, votedFor=leader302 02:31:50,274 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower303 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:31:50,274 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower303: Last sync leader does not match current leader leader302, need to catch up to -1 02:31:50,274 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower303 updated: matchIndex: -1, nextIndex: 0 02:31:50,274 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower303: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:50.274] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower303] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower303, initialSyncDone=false] 02:31:50,275 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower305: Set currentTerm=2, votedFor=null 02:31:50,275 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower305: Last sync leader does not match current leader leader302, need to catch up to -1 02:31:50,275 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader302 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower305 updated: matchIndex: -1, nextIndex: 0 02:31:50,275 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower305: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:50.275] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower305] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower305, initialSyncDone=false] 02:31:50,308 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:31:50,308 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 02:31:50,308 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 02:31:50,308 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower304 starting 02:31:50,308 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower304: onShutDown 02:31:50,309 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower304: terminating thread VirtualThread[#713,follower304-writer-141]/waiting 02:31:50,309 PM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: received 1 entries in 479.8 ms 02:31:50,309 PM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: persisted 1 entries in 71.28 μs 02:31:50,309 PM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@24063a3c stopped 02:31:50,309 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower304: journal closed 02:31:50,309 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower304: journal closed 02:31:50,309 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower304: stopped executor with 0 remaining tasks 02:31:50,309 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower304: stopped executor with 0 remaining tasks 02:31:50,309 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower304 ending 02:31:50,309 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 02:31:50,309 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 02:31:50,310 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader302: onRequestLeadership RequestLeadership [requestedFollowerId=follower304, replyTo=Actor[pekko://test/user/$b#-1946034304]] 02:31:50,310 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader302: Initiating leader transfer 02:31:50,310 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower303: onLeaderTransitioning: LeaderTransitioning [leaderId=leader302] 02:31:50,310 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader302: pauseLeader successfully completed - doing transfer 02:31:50,310 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower305: onLeaderTransitioning: LeaderTransitioning [leaderId=leader302] 02:31:50,310 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader302 (Leader): Attempting to transfer leadership 02:31:50,310 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower304: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:50.310] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: LeaderTransitioning [leaderId=leader302] 02:31:50,310 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower303: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:50,310 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower305: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:50.310] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:31:50,310 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower303: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) [WARN] [06/27/2025 14:31:50.311] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower303] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower303, initialSyncDone=true] 02:31:50,311 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower305: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) [WARN] [06/27/2025 14:31:50.311] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower305] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower305, initialSyncDone=true] [WARN] [06/27/2025 14:31:50.389] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:31:50,510 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower303: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 02:31:50,510 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower305: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) [WARN] [06/27/2025 14:31:50.510] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:31:50,630 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader302: leader transfer aborted 02:31:50,630 PM [test-pekko.actor.default-dispatcher-16] [WARN] RaftActorLeadershipTransferCohort - leader302: Failed to transfer leadership in 320.2 ms 02:31:50,630 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader302: LeadershipTransfer request from follower304 failed 02:31:50,631 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Leader - leader302 (Leader): Leadership transfer expired [WARN] [06/27/2025 14:31:50.631] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:31:50,631 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower303: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 02:31:50,631 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower305: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 02:31:50,663 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 02:31:50,663 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower303-notifier306#736948145] 02:31:50,674 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower303-collector307#1877284851] 02:31:50,674 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower303] 02:31:50,674 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower303: terminating thread VirtualThread[#712,follower303-writer-140]/waiting 02:31:50,674 PM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: received 1 entries in 849.9 ms 02:31:50,674 PM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: persisted 1 entries in 50.78 μs 02:31:50,674 PM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@70bd4249 stopped 02:31:50,675 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower303: journal closed 02:31:50,675 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower303: journal closed 02:31:50,675 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower303: stopped executor with 0 remaining tasks 02:31:50,675 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower303: stopped executor with 0 remaining tasks 02:31:50,675 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower304-notifier308#-1063688720] 02:31:50,675 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower304-collector309#-614290009] 02:31:50,675 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower304] 02:31:50,675 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower305-notifier310#618128916] 02:31:50,676 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower305-collector311#-1566889364] 02:31:50,676 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower305] 02:31:50,676 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower305: terminating thread VirtualThread[#714,follower305-writer-142]/waiting 02:31:50,676 PM [follower305-writer-142] [DEBUG] JournalWriteTask - follower305: received 1 entries in 840.7 ms 02:31:50,676 PM [follower305-writer-142] [DEBUG] JournalWriteTask - follower305: persisted 1 entries in 26.57 μs 02:31:50,676 PM [follower305-writer-142] [DEBUG] JournalWriteTask - follower305: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3471f8e7 stopped 02:31:50,676 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower305: journal closed 02:31:50,676 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower305: journal closed 02:31:50,676 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower305: stopped executor with 0 remaining tasks 02:31:50,676 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower305: stopped executor with 0 remaining tasks 02:31:50,677 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader302-notifier312#239552491] 02:31:50,677 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader302-collector313#-133050618] 02:31:50,677 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader302] 02:31:50,677 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - leader302: terminating thread VirtualThread[#715,leader302-writer-143]/waiting 02:31:50,677 PM [leader302-writer-143] [DEBUG] JournalWriteTask - leader302: received 1 entries in 835.6 ms 02:31:50,677 PM [leader302-writer-143] [DEBUG] JournalWriteTask - leader302: persisted 1 entries in 30.53 μs 02:31:50,677 PM [leader302-writer-143] [DEBUG] JournalWriteTask - leader302: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@1d5c7510 stopped 02:31:50,677 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader302: journal closed 02:31:50,678 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - leader302: journal closed 02:31:50,678 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: stopped executor with 0 remaining tasks 02:31:50,678 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader302: stopped executor with 0 remaining tasks 02:31:50,678 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$b#-1946034304] 02:31:50,680 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 02:31:50,680 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/27/2025 14:31:50.682] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower315]: org.opendaylight.controller.cluster.common.actor.Monitor@6dac5169 02:31:50,683 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower315: Starting recovery with journal batch size 1000 02:31:50,683 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: started executor 02:31:50,683 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: started executor 02:31:50,683 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower315: metadata journal open: replayFrom=1 applyTo=0 02:31:50,684 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower315: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:50,684 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower315: journal open: applyTo=0 02:31:50,684 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@769d3f8 started 02:31:50,684 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: waiting for more work 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower315/TermInfo.properties does not exist 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower315/journal-v1-1.log 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower315/metadata-v1-1.log 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: no eligible files found 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower315: SnapshotOffer called. 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower315: initializing from snapshot taken at 2025-06-27T14:31:50.680Z 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=1, votedFor=null 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower315: Updated server config: isVoting: true, peers: [PeerInfo [id=follower317, address=pekko://test/user/follower317, votingState=NON_VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=VOTING], PeerInfo [id=leader314, address=pekko://test/user/leader314, votingState=VOTING]] 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - Recovery snapshot applied for follower315 in 83.85 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower315: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:50.680Z 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower315: support() called 02:31:50,685 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower315/snapshot-00000000685EAB56-2887FA00.tmp 02:31:50,687 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower316: Starting recovery with journal batch size 1000 02:31:50,687 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: started executor 02:31:50,687 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: started executor 02:31:50,688 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower316: metadata journal open: replayFrom=1 applyTo=0 02:31:50,688 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower316: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:50,688 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower316: journal open: applyTo=0 02:31:50,688 PM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@12abcef3 started 02:31:50,688 PM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: waiting for more work [WARN] [06/27/2025 14:31:50.688] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower316]: org.opendaylight.controller.cluster.common.actor.Monitor@7644fa00 02:31:50,689 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower315/snapshot-00000000685EAB56-2887FA00.v1 02:31:50,689 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower315/journal-v1-1.log 02:31:50,689 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower315/snapshot-00000000685EAB56-2887FA00.v1 02:31:50,689 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower315/metadata-v1-1.log 02:31:50,689 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower315: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:50.680Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower315/snapshot-00000000685EAB56-2887FA00.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:50,689 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower315: local snapshot saved in 4.025 ms, deleting Pekko-persisted snapshots 02:31:50,689 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower316/TermInfo.properties does not exist 02:31:50,689 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower316/journal-v1-1.log 02:31:50,689 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower316/metadata-v1-1.log 02:31:50,689 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: no eligible files found 02:31:50,689 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower316: SnapshotOffer called. 02:31:50,689 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower316: initializing from snapshot taken at 2025-06-27T14:31:50.680Z 02:31:50,689 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=1, votedFor=null [WARN] [06/27/2025 14:31:50.690] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower317]: org.opendaylight.controller.cluster.common.actor.Monitor@23eb912b 02:31:50,690 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - follower316: Updated server config: isVoting: true, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=follower317, address=pekko://test/user/follower317, votingState=NON_VOTING], PeerInfo [id=leader314, address=pekko://test/user/leader314, votingState=VOTING]] 02:31:50,690 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for follower316 in 105.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:50,690 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower316: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:50.680Z 02:31:50,690 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower316: support() called 02:31:50,690 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower315: Recovery completed in 5.026 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:50,690 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower316/snapshot-00000000685EAB56-2887FA00.tmp 02:31:50,690 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower317: Starting recovery with journal batch size 1000 02:31:50,690 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower317: started executor 02:31:50,690 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower317: started executor 02:31:50,691 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower317: metadata journal open: replayFrom=1 applyTo=0 02:31:50,691 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower317: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:50,691 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower317: journal open: applyTo=0 02:31:50,691 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@2926aa81 started 02:31:50,692 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: waiting for more work 02:31:50,692 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower317/TermInfo.properties does not exist 02:31:50,692 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower316/snapshot-00000000685EAB56-2887FA00.v1 02:31:50,693 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower316/journal-v1-1.log 02:31:50,693 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower316/snapshot-00000000685EAB56-2887FA00.v1 02:31:50,693 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower316/metadata-v1-1.log 02:31:50,693 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:50.680Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower316/snapshot-00000000685EAB56-2887FA00.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:50,693 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower316: local snapshot saved in 2.933 ms, deleting Pekko-persisted snapshots [WARN] [06/27/2025 14:31:50.693] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader314]: org.opendaylight.controller.cluster.common.actor.Monitor@4b561c6b 02:31:50,693 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower316: Recovery completed in 3.677 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:50,693 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=1, votedFor=null 02:31:50,693 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower315: Local TermInfo store seeded with TermInfo{term=1} 02:31:50,693 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower315: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@4375af9b 02:31:50,693 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower315: Moving last applied index from -1 to -1 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower317/journal-v1-1.log 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower317/metadata-v1-1.log 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: no eligible files found 02:31:50,694 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower315: journal recovery completedin 425.4 μs with journalIndex=1 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower317: SnapshotOffer called. 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower317: initializing from snapshot taken at 2025-06-27T14:31:50.680Z 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=1, votedFor=null 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower317: Updated server config: isVoting: false, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=VOTING], PeerInfo [id=leader314, address=pekko://test/user/leader314, votingState=VOTING]] 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - Recovery snapshot applied for follower317 in 50.99 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower317: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:50.680Z 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower317: support() called 02:31:50,694 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - leader314: Starting recovery with journal batch size 1000 02:31:50,694 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: started executor 02:31:50,694 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: started executor 02:31:50,694 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower317/snapshot-00000000685EAB56-2887FA00.tmp 02:31:50,694 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader314: metadata journal open: replayFrom=1 applyTo=0 02:31:50,695 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower315: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:50,695 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader314: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:50,695 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - leader314: journal open: applyTo=0 02:31:50,695 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@572cbe40 started 02:31:50,695 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: waiting for more work 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/leader314/TermInfo.properties does not exist 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/leader314/journal-v1-1.log 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/leader314/metadata-v1-1.log 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: no eligible files found 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - leader314: SnapshotOffer called. 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - leader314: initializing from snapshot taken at 2025-06-27T14:31:50.680Z 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=1, votedFor=null 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - leader314: Updated server config: isVoting: true, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=follower317, address=pekko://test/user/follower317, votingState=NON_VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=VOTING]] 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - Recovery snapshot applied for leader314 in 61.03 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - leader314: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:50.680Z 02:31:50,696 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader314: support() called 02:31:50,696 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=1, votedFor=null 02:31:50,696 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower316: Local TermInfo store seeded with TermInfo{term=1} 02:31:50,696 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower316: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@1cefdd91 02:31:50,696 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower316: Moving last applied index from -1 to -1 02:31:50,697 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/leader314/snapshot-00000000685EAB56-2887FA00.tmp 02:31:50,697 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower316: journal recovery completedin 222.0 μs with journalIndex=1 02:31:50,697 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower316: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:50,698 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower317/snapshot-00000000685EAB56-2887FA00.v1 02:31:50,698 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower317/journal-v1-1.log 02:31:50,698 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower317/snapshot-00000000685EAB56-2887FA00.v1 02:31:50,698 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower317/metadata-v1-1.log 02:31:50,698 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:50.680Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/follower317/snapshot-00000000685EAB56-2887FA00.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:50,698 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower317: local snapshot saved in 3.932 ms, deleting Pekko-persisted snapshots 02:31:50,699 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/leader314/snapshot-00000000685EAB56-2887FA00.v1 02:31:50,699 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower317: Recovery completed in 4.893 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:50,699 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/leader314/journal-v1-1.log 02:31:50,699 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/leader314/snapshot-00000000685EAB56-2887FA00.v1 02:31:50,699 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/leader314/metadata-v1-1.log 02:31:50,699 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:50.680Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9677212092508700652/leader314/snapshot-00000000685EAB56-2887FA00.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:50,699 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - leader314: local snapshot saved in 2.625 ms, deleting Pekko-persisted snapshots 02:31:50,700 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader314: Recovery completed in 3.392 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:50,702 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=1, votedFor=null 02:31:50,702 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower317: Local TermInfo store seeded with TermInfo{term=1} 02:31:50,702 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower317: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@1770ffd2 02:31:50,702 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower317: Moving last applied index from -1 to -1 02:31:50,702 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower317: journal recovery completedin 409.8 μs with journalIndex=1 02:31:50,702 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=1, votedFor=null 02:31:50,702 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader314: Local TermInfo store seeded with TermInfo{term=1} 02:31:50,702 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader314: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@f6ecdaa 02:31:50,703 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader314: Moving last applied index from -1 to -1 02:31:50,703 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - leader314: journal recovery completedin 263.2 μs with journalIndex=1 02:31:50,703 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower317: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:50,703 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader314: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:51,39 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:31:51,39 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - leader314 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.8:17355, Removed) 02:31:51,39 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - leader314 (Follower): Received ElectionTimeout - switching to Candidate 02:31:51,40 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader314 (Candidate): Election: Candidate has following voting peers: [follower315, follower316] 02:31:51,42 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=2, votedFor=leader314 02:31:51,43 PM [test-pekko.actor.default-dispatcher-11] [INFO] Candidate - leader314 (Candidate): Starting new election term 2 02:31:51,43 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader314 (Candidate): Sending RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} to peer follower315 02:31:51,43 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader314 (Candidate): Sending RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} to peer follower316 02:31:51,43 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader314 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:31:51,43 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower315 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:51,43 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower315 (Follower): Cluster state: [] 02:31:51,43 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower316 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:51,43 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower315 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:51,43 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower315 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:51,43 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower316 (Follower): Cluster state: [] 02:31:51,43 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower316 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:51,43 PM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower316 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:51,47 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=2, votedFor=null 02:31:51,47 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower315 (Follower): In requestVote: RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:51,48 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=2, votedFor=null 02:31:51,48 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower316 (Follower): In requestVote: RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:51,50 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=2, votedFor=leader314 02:31:51,50 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower315 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:31:51,50 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader314 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:31:51,50 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader314 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:31:51,51 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader314 (Leader): Election: Leader has following peers: [follower315, follower317, follower316] 02:31:51,51 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:51,51 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:51,51 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:51,51 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - leader314 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:31:51,51 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower317 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:31:51,51 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower315: Last sync leader does not match current leader leader314, need to catch up to -1 02:31:51,51 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=2, votedFor=leader314 02:31:51,51 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower316 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:31:51,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower315 updated: matchIndex: -1, nextIndex: 0 02:31:51,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:51,52 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower316: Last sync leader does not match current leader leader314, need to catch up to -1 02:31:51,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower316 updated: matchIndex: -1, nextIndex: 0 02:31:51,52 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:51.052] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower315] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower315, initialSyncDone=false] [WARN] [06/27/2025 14:31:51.052] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=false] 02:31:51,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=2, votedFor=null 02:31:51,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower317: Last sync leader does not match current leader leader314, need to catch up to -1 02:31:51,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower317 updated: matchIndex: -1, nextIndex: 0 02:31:51,53 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:51.054] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=false] 02:31:51,56 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:31:51,56 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader314: Persist data index=0 term=2 command=zero 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader314: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower315 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower315: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower317 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower317: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 02:31:51,56 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): Number of entries to be appended = 1 02:31:51,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 02:31:51,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): Append entry to log zero 02:31:51,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower315: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 02:31:51,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): Log size is now 1 02:31:51,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower315, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:31:51,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached -1 (of expected -1) 02:31:51,57 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryStoreCompleter - follower315: awaiting more completions to resolve 1 deferred callback(s) 02:31:51,57 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: received 1 entries in 361.3 ms 02:31:51,57 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower317 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 02:31:51,57 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower317 (Follower): Number of entries to be appended = 1 02:31:51,57 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower317 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 02:31:51,57 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower317 (Follower): Append entry to log zero 02:31:51,57 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower317: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 02:31:51,57 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower317 (Follower): Log size is now 1 02:31:51,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower315 updated: matchIndex: 0, nextIndex: 1 02:31:51,57 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: received 1 entries in 373.3 ms 02:31:51,57 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower317 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower317, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:31:51,57 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached -1 (of expected -1) 02:31:51,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,57 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryStoreCompleter - follower317: awaiting more completions to resolve 1 deferred callback(s) 02:31:51,57 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: received 1 entries in 365.7 ms 02:31:51,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower317 updated: matchIndex: 0, nextIndex: 1 02:31:51,57 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,63 PM [leader314-writer-147] [DEBUG] EntryStoreCompleter - leader314: 1 completion(s) pending 02:31:51,63 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: persisted 1 entries in 5.758 ms 02:31:51,63 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: waiting for more work 02:31:51,63 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryStoreCompleter - leader314: running 1 completion(s) 02:31:51,63 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader314: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} 02:31:51,63 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 02:31:51,63 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - leader314 (Leader): Setting last applied to 0 02:31:51,63 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader314: Moving last applied index from -1 to 0 02:31:51,63 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader314: Applying state for log index 0 data zero 02:31:51,63 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader314: applyState called: zero 02:31:51,63 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: received 1 entries in 604.1 μs 02:31:51,63 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: persisted 1 entries in 123.0 μs 02:31:51,63 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: waiting for more work 02:31:51,64 PM [follower317-writer-146] [DEBUG] EntryStoreCompleter - follower317: 1 completion(s) pending 02:31:51,64 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: persisted 1 entries in 6.337 ms 02:31:51,64 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: waiting for more work 02:31:51,64 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryStoreCompleter - follower317: running 1 completion(s) 02:31:51,64 PM [follower315-writer-144] [DEBUG] EntryStoreCompleter - follower315: 1 completion(s) pending 02:31:51,64 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower317: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 02:31:51,64 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryStoreCompleter - follower315: running 1 completion(s) 02:31:51,64 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryStoreCompleter - follower317: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c087fa30@742ebf10 02:31:51,64 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower315: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 02:31:51,64 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryStoreCompleter - follower315: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c087fa30@3191b45 02:31:51,64 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: persisted 1 entries in 6.459 ms 02:31:51,64 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: waiting for more work [WARN] [06/27/2025 14:31:51.064] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=true] [WARN] [06/27/2025 14:31:51.064] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower315] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower315, initialSyncDone=true] 02:31:51,170 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 02:31:51,170 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 02:31:51,170 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower317 (Follower): Commit index set to 0 02:31:51,170 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 02:31:51,170 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower317 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 02:31:51,170 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower317 (Follower): Setting last applied to 0 02:31:51,170 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower317: Moving last applied index from -1 to 0 02:31:51,170 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower317: Applying state for log index 0 data zero 02:31:51,170 PM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower317: applyState called: zero 02:31:51,171 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower315 (Follower): Commit index set to 0 02:31:51,171 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 02:31:51,171 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower315 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 02:31:51,171 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower315 (Follower): Setting last applied to 0 02:31:51,171 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: received 1 entries in 106.9 ms 02:31:51,171 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower315: Moving last applied index from -1 to 0 02:31:51,171 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower315: Applying state for log index 0 data zero 02:31:51,171 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower315: applyState called: zero 02:31:51,171 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: persisted 1 entries in 211.8 μs 02:31:51,171 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: waiting for more work 02:31:51,171 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: received 1 entries in 107.2 ms 02:31:51,171 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: persisted 1 entries in 126.1 μs 02:31:51,171 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: waiting for more work 02:31:51,207 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 02:31:51,207 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 02:31:51,207 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader314: onShutDown 02:31:51,207 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader314: Initiating leader transfer 02:31:51,208 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower315: onLeaderTransitioning: LeaderTransitioning [leaderId=leader314] 02:31:51,208 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader314: pauseLeader successfully completed - doing transfer 02:31:51,208 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader314 (Leader): Attempting to transfer leadership 02:31:51,208 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,208 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower317: onLeaderTransitioning: LeaderTransitioning [leaderId=leader314] 02:31:51,208 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,208 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,208 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 02:31:51,208 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 02:31:51,208 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Leader - leader314 (Leader): tryToCompleteLeadershipTransfer: followerId: follower317, matchIndex: 0, lastIndex: 0, isVoting: false 02:31:51,208 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 02:31:51,208 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Leader - leader314 (Leader): tryToCompleteLeadershipTransfer: followerId: follower315, matchIndex: 0, lastIndex: 0, isVoting: true 02:31:51,208 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Leader - leader314 (Leader): Follower's log matches - sending ElectionTimeout 02:31:51,208 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,209 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,209 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,209 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 02:31:51,209 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 02:31:51,209 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Leader - leader314 (Leader): Leader transfer complete 02:31:51,209 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader314: leader transfer complete - waiting for new leader 02:31:51,209 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 02:31:51,209 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:31:51,209 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower315 (Follower): Received TimeoutNow - switching to Candidate 02:31:51,209 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower315 (Candidate): Election: Candidate has following voting peers: [follower316, leader314] 02:31:51,214 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=3, votedFor=follower315 02:31:51,214 PM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - follower315 (Candidate): Starting new election term 3 02:31:51,214 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower315 (Candidate): Sending RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} to peer follower316 02:31:51,214 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower315 (Candidate): Sending RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} to peer leader314 02:31:51,214 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower316 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:51,214 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - follower315 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 02:31:51,214 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower316 (Follower): Cluster state: [] 02:31:51,214 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower316 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 02:31:51,214 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower316 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2}" message is greater than follower's term 2 - updating term 02:31:51,214 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader314 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:51,214 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader314 (Leader): Cluster state: [] 02:31:51,214 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader314 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 02:31:51,214 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractLeader - leader314 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2}" message is greater than leader's term 2 - switching to Follower 02:31:51,217 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=3, votedFor=null 02:31:51,217 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader314 (Leader): Leadership transfer in progress - processing RequestVote 02:31:51,217 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader314 (Leader): In requestVote: RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 02:31:51,218 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=3, votedFor=null 02:31:51,218 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower316 (Follower): In requestVote: RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:51,220 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=3, votedFor=follower315 02:31:51,220 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader314 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 02:31:51,220 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - follower315 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 02:31:51,220 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - follower315 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 02:31:51,220 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower315 (Leader): Election: Leader has following peers: [follower317, follower316, leader314] 02:31:51,220 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower317: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,220 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower316: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,220 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower leader314: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,220 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower317 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower315, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 02:31:51,220 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - follower315 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 02:31:51,220 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - leader314 (Leader) :- Switching from behavior Leader to Follower, election term: 3 02:31:51,221 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower315 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader314 updated: matchIndex: 0, nextIndex: 1 02:31:51,221 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower leader314: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,221 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - leader314: Last sync leader does not match current leader follower315, need to catch up to -1 02:31:51,221 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader314: leader changed to follower315 02:31:51,221 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorLeadershipTransferCohort - leader314: Successfully transferred leadership to follower315 in 13.35 ms 02:31:51,221 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader314: leader transfer succeeded - sending PoisonPill 02:31:51,221 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - leader314: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) 02:31:51,221 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=3, votedFor=follower315 02:31:51,221 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower316 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} [WARN] [06/27/2025 14:31:51.221] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader314] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader314, initialSyncDone=false] 02:31:51,221 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRaftStorage - leader314: terminating thread VirtualThread[#719,leader314-writer-147]/waiting 02:31:51,222 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: received 1 entries in 158.2 ms 02:31:51,222 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: persisted 1 entries in 28.30 μs 02:31:51,222 PM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@572cbe40 stopped 02:31:51,222 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader314: journal closed 02:31:51,222 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - leader314: journal closed 02:31:51,222 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: stopped executor with 0 remaining tasks 02:31:51,222 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader314: stopped executor with 0 remaining tasks [WARN] [06/27/2025 14:31:51.222] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/leader314] received dead letter: FollowerInitialSyncUpStatus[memberId=leader314, initialSyncDone=true] 02:31:51,223 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=3, votedFor=null 02:31:51,223 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower317: Last sync leader does not match current leader follower315, need to catch up to -1 02:31:51,223 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower317 updated: matchIndex: 0, nextIndex: 1 02:31:51,223 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:51.223] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=false] 02:31:51,223 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) [WARN] [06/27/2025 14:31:51.223] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=true] 02:31:51,740 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) 02:31:51,740 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower316: Last sync leader does not match current leader follower315, need to catch up to -1 [WARN] [06/27/2025 14:31:51.740] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader314] received dead letter from TestActor[pekko://test/user/follower315]: AppendEntries{term=3, leaderId=follower315, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [06/27/2025 14:31:51.740] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=false] 02:31:51,740 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower315 (Leader): sendAppendEntries: 0 is present for follower follower316 02:31:51,740 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower315 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=3, leaderId=follower315, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower316 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=follower315, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower316 (Follower): Number of entries to be appended = 1 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower316 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower316 (Follower): Append entry to log zero 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower316: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=false} 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower316 (Follower): Log size is now 1 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower316 (Follower): Commit index set to 0 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower316 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower316, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower316: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower316 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower316 (Follower): Setting last applied to 0 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower316: Moving last applied index from -1 to 0 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower316: Applying state for log index 0 data zero 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower316: applyState called: zero 02:31:51,741 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower316 updated: matchIndex: 0, nextIndex: 1 02:31:51,741 PM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: received 1 entries in 1.053 s 02:31:51,741 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:51,741 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryStoreCompleter - follower316: awaiting more completions to resolve 1 deferred callback(s) 02:31:51,746 PM [follower316-writer-145] [DEBUG] EntryStoreCompleter - follower316: 1 completion(s) pending 02:31:51,746 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryStoreCompleter - follower316: running 1 completion(s) 02:31:51,746 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower316: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 02:31:51,746 PM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: persisted 1 entries in 4.511 ms 02:31:51,746 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryStoreCompleter - follower316: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c087fa30@5975c70d 02:31:51,746 PM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: received 1 entries in 10.10 μs 02:31:51,746 PM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: persisted 1 entries in 186.5 μs 02:31:51,746 PM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: waiting for more work [WARN] [06/27/2025 14:31:51.746] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=true] 02:31:51,762 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower316: onLeaderTransitioning: LeaderTransitioning [leaderId=leader314] 02:31:51,762 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 02:31:51,762 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower316 starting 02:31:51,762 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower316: onShutDown 02:31:51,763 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower316: terminating thread VirtualThread[#717,follower316-writer-145]/waiting 02:31:51,763 PM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: received 1 entries in 16.77 ms 02:31:51,763 PM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: persisted 1 entries in 41.20 μs 02:31:51,763 PM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@12abcef3 stopped 02:31:51,763 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower316: journal closed 02:31:51,763 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower316: journal closed 02:31:51,763 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: stopped executor with 0 remaining tasks 02:31:51,763 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: stopped executor with 0 remaining tasks 02:31:51,763 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower316 ending 02:31:51,763 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 02:31:51,763 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower315-notifier318#-436701040] 02:31:51,774 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower315-collector319#-541368714] 02:31:51,775 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower315] 02:31:51,775 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower315: terminating thread VirtualThread[#716,follower315-writer-144]/waiting 02:31:51,775 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: received 1 entries in 603.8 ms 02:31:51,775 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: persisted 1 entries in 50.58 μs 02:31:51,775 PM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@769d3f8 stopped 02:31:51,775 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower315: journal closed 02:31:51,775 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower315: journal closed 02:31:51,775 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: stopped executor with 0 remaining tasks 02:31:51,775 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: stopped executor with 0 remaining tasks 02:31:51,775 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower316-notifier320#2142484527] 02:31:51,776 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower316-collector321#1293209028] 02:31:51,776 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower316] 02:31:51,776 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower317-notifier322#-1406568922] 02:31:51,776 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower317-collector323#1660720403] 02:31:51,776 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower317] 02:31:51,777 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower317: terminating thread VirtualThread[#718,follower317-writer-146]/waiting 02:31:51,777 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: received 1 entries in 605.8 ms 02:31:51,777 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: persisted 1 entries in 33.75 μs 02:31:51,777 PM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@2926aa81 stopped 02:31:51,777 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower317: journal closed 02:31:51,777 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower317: journal closed 02:31:51,777 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower317: stopped executor with 0 remaining tasks 02:31:51,777 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower317: stopped executor with 0 remaining tasks 02:31:51,777 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader314-notifier324#-1511548267] 02:31:51,780 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader314-collector325#161565336] 02:31:51,780 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader314] 02:31:51,783 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [06/27/2025 14:31:51.785] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader326]: org.opendaylight.controller.cluster.common.actor.Monitor@f9a54dc 02:31:51,785 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - leader326: Starting recovery with journal batch size 1000 02:31:51,785 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader326: started executor 02:31:51,785 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader326: started executor 02:31:51,786 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader326: metadata journal open: replayFrom=1 applyTo=0 02:31:51,786 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader326: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:51,786 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - leader326: journal open: applyTo=0 02:31:51,786 PM [leader326-writer-148] [DEBUG] JournalWriteTask - leader326: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@106a3641 started 02:31:51,786 PM [leader326-writer-148] [DEBUG] JournalWriteTask - leader326: waiting for more work 02:31:51,788 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest4002043011708870551/leader326/TermInfo.properties does not exist 02:31:51,788 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader326: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest4002043011708870551/leader326/journal-v1-1.log 02:31:51,788 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader326: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest4002043011708870551/leader326/metadata-v1-1.log 02:31:51,788 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader326: no eligible files found 02:31:51,788 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader326: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:51,791 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader326: Set currentTerm=0, votedFor=null 02:31:51,791 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader326: Local TermInfo store seeded with TermInfo{term=0} 02:31:51,791 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader326: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@1416a6d9 02:31:51,791 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader326: Moving last applied index from -1 to -1 02:31:51,791 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - leader326: journal recovery completedin 285.1 μs with journalIndex=1 02:31:51,792 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:31:51,792 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - leader326 (Follower): Received TimeoutNow - switching to Candidate 02:31:51,792 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader326 (Candidate): Election: Candidate has following voting peers: [] 02:31:51,795 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader326: Set currentTerm=1, votedFor=leader326 02:31:51,795 PM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader326 (Candidate): Starting new election term 1 02:31:51,795 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - leader326 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 02:31:51,796 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader326 (Candidate): Received ElectionTimeout 02:31:51,796 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader326 (Leader): Election: Leader has following peers: [] 02:31:51,796 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - leader326 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 02:31:51,796 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader326 starting 02:31:51,796 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader326: onShutDown 02:31:51,796 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - leader326: terminating thread VirtualThread[#720,leader326-writer-148]/waiting 02:31:51,797 PM [leader326-writer-148] [DEBUG] JournalWriteTask - leader326: received 1 entries in 10.28 ms 02:31:51,797 PM [leader326-writer-148] [DEBUG] JournalWriteTask - leader326: persisted 1 entries in 205.0 μs 02:31:51,797 PM [leader326-writer-148] [DEBUG] JournalWriteTask - leader326: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@106a3641 stopped 02:31:51,797 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader326: journal closed 02:31:51,797 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - leader326: journal closed 02:31:51,797 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader326: stopped executor with 0 remaining tasks 02:31:51,797 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader326: stopped executor with 0 remaining tasks 02:31:51,797 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader326 ending 02:31:51,797 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 02:31:51,797 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader326-collector330#-2128642231] 02:31:51,808 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader326] 02:31:51,810 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 02:31:51,810 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/27/2025 14:31:51.812] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower332]: org.opendaylight.controller.cluster.common.actor.Monitor@696bbe3e 02:31:51,813 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower332: Starting recovery with journal batch size 1000 02:31:51,814 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower332: started executor 02:31:51,814 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower332: started executor 02:31:51,814 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower332: metadata journal open: replayFrom=1 applyTo=0 02:31:51,815 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower332: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:51,815 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower332: journal open: applyTo=0 02:31:51,815 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@5d419f9d started 02:31:51,815 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: waiting for more work 02:31:51,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower332/TermInfo.properties does not exist 02:31:51,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower332/journal-v1-1.log 02:31:51,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower332/metadata-v1-1.log 02:31:51,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower332: no eligible files found 02:31:51,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower332: SnapshotOffer called. 02:31:51,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower332: initializing from snapshot taken at 2025-06-27T14:31:51.810Z 02:31:51,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=1, votedFor=null 02:31:51,816 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - follower332: Updated server config: isVoting: true, peers: [PeerInfo [id=leader331, address=pekko://test/user/leader331, votingState=VOTING], PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=VOTING], PeerInfo [id=follower334, address=pekko://test/user/follower334, votingState=NON_VOTING]] 02:31:51,816 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - Recovery snapshot applied for follower332 in 126.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:51,817 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower332: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:51.810Z 02:31:51,817 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower332: support() called 02:31:51,817 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower332: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower332/snapshot-00000000685EAB57-30479E80.tmp [WARN] [06/27/2025 14:31:51.817] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower333]: org.opendaylight.controller.cluster.common.actor.Monitor@6459f81d 02:31:51,819 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower333: Starting recovery with journal batch size 1000 02:31:51,819 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: started executor 02:31:51,819 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: started executor 02:31:51,819 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower333: metadata journal open: replayFrom=1 applyTo=0 02:31:51,820 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower332: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower332/snapshot-00000000685EAB57-30479E80.v1 02:31:51,820 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower332: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower332/snapshot-00000000685EAB57-30479E80.v1 02:31:51,820 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower332/journal-v1-1.log 02:31:51,820 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower332/metadata-v1-1.log 02:31:51,820 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower332: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:51.810Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower332/snapshot-00000000685EAB57-30479E80.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:51,820 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower332: local snapshot saved in 3.469 ms, deleting Pekko-persisted snapshots 02:31:51,820 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower333: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:51,820 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower333: journal open: applyTo=0 02:31:51,821 PM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3f728ca1 started 02:31:51,821 PM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: waiting for more work 02:31:51,821 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower332: Recovery completed in 4.726 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:51,821 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower333/TermInfo.properties does not exist 02:31:51,821 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower333/journal-v1-1.log 02:31:51,821 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower333/metadata-v1-1.log 02:31:51,822 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: no eligible files found 02:31:51,822 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower333: SnapshotOffer called. 02:31:51,822 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower333: initializing from snapshot taken at 2025-06-27T14:31:51.810Z 02:31:51,822 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=1, votedFor=null 02:31:51,822 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower333: Updated server config: isVoting: true, peers: [PeerInfo [id=leader331, address=pekko://test/user/leader331, votingState=VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING], PeerInfo [id=follower334, address=pekko://test/user/follower334, votingState=NON_VOTING]] [WARN] [06/27/2025 14:31:51.822] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower334]: org.opendaylight.controller.cluster.common.actor.Monitor@a1dcb90 02:31:51,822 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower334: Starting recovery with journal batch size 1000 02:31:51,823 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: started executor 02:31:51,823 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: started executor 02:31:51,823 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - Recovery snapshot applied for follower333 in 930.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:51,823 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower333: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:51.810Z 02:31:51,823 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower333: support() called 02:31:51,823 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower334: metadata journal open: replayFrom=1 applyTo=0 02:31:51,823 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower333/snapshot-00000000685EAB57-30479E80.tmp 02:31:51,823 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower334: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:51,823 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower334: journal open: applyTo=0 02:31:51,823 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=1, votedFor=null 02:31:51,823 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower332: Local TermInfo store seeded with TermInfo{term=1} 02:31:51,823 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower332: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@c1a24d4 02:31:51,823 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower332: Moving last applied index from -1 to -1 02:31:51,823 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@129123de started 02:31:51,823 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: waiting for more work 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower334/TermInfo.properties does not exist 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower334/journal-v1-1.log 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower334/metadata-v1-1.log 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: no eligible files found 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower334: SnapshotOffer called. 02:31:51,824 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower332: journal recovery completedin 326.8 μs with journalIndex=1 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower334: initializing from snapshot taken at 2025-06-27T14:31:51.810Z 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower334: Set currentTerm=1, votedFor=null 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - follower334: Updated server config: isVoting: false, peers: [PeerInfo [id=leader331, address=pekko://test/user/leader331, votingState=VOTING], PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING]] 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for follower334 in 51.56 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower334: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:51.810Z 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower334: support() called 02:31:51,824 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower334/snapshot-00000000685EAB57-30479E80.tmp [WARN] [06/27/2025 14:31:51.825] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader331]: org.opendaylight.controller.cluster.common.actor.Monitor@10dd067a 02:31:51,825 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - leader331: Starting recovery with journal batch size 1000 02:31:51,824 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower332: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:51,825 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: started executor 02:31:51,826 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: started executor 02:31:51,826 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader331: metadata journal open: replayFrom=1 applyTo=0 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower334/snapshot-00000000685EAB57-30479E80.v1 02:31:51,827 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader331: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:51,827 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - leader331: journal open: applyTo=0 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower334/snapshot-00000000685EAB57-30479E80.v1 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower334/journal-v1-1.log 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower334/metadata-v1-1.log 02:31:51,827 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@5308d7f3 started 02:31:51,827 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: waiting for more work 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower334: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:51.810Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower334/snapshot-00000000685EAB57-30479E80.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower334: local snapshot saved in 2.828 ms, deleting Pekko-persisted snapshots 02:31:51,827 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower333/snapshot-00000000685EAB57-30479E80.v1 02:31:51,827 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower333/snapshot-00000000685EAB57-30479E80.v1 02:31:51,827 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower333/journal-v1-1.log 02:31:51,827 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower333/metadata-v1-1.log 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/leader331/TermInfo.properties does not exist 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/leader331/journal-v1-1.log 02:31:51,827 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:51.810Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/follower333/snapshot-00000000685EAB57-30479E80.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/leader331/metadata-v1-1.log 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader331: no eligible files found 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - leader331: SnapshotOffer called. 02:31:51,827 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower333: local snapshot saved in 4.571 ms, deleting Pekko-persisted snapshots 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - leader331: initializing from snapshot taken at 2025-06-27T14:31:51.810Z 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader331: Set currentTerm=1, votedFor=null 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - leader331: Updated server config: isVoting: true, peers: [PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING], PeerInfo [id=follower334, address=pekko://test/user/follower334, votingState=NON_VOTING]] 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for leader331 in 38.70 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - leader331: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:51.810Z 02:31:51,827 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader331: support() called 02:31:51,827 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower334: Recovery completed in 3.569 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:51,828 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader331: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/leader331/snapshot-00000000685EAB57-30479E80.tmp 02:31:51,828 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower333: Recovery completed in 6.047 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:51,831 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower334: Set currentTerm=1, votedFor=null 02:31:51,831 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower334: Local TermInfo store seeded with TermInfo{term=1} 02:31:51,831 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower334: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@289a2316 02:31:51,831 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractReplicatedLog - follower334: Moving last applied index from -1 to -1 02:31:51,831 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] JournalRecovery - follower334: journal recovery completedin 242.9 μs with journalIndex=1 02:31:51,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=1, votedFor=null 02:31:51,832 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower333: Local TermInfo store seeded with TermInfo{term=1} 02:31:51,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower333: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@aa629bb 02:31:51,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower333: Moving last applied index from -1 to -1 02:31:51,832 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower334: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:51,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower333: journal recovery completedin 117.2 μs with journalIndex=1 02:31:51,832 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader331: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/leader331/snapshot-00000000685EAB57-30479E80.v1 02:31:51,832 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader331: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/leader331/snapshot-00000000685EAB57-30479E80.v1 02:31:51,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower333: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:51,832 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/leader331/journal-v1-1.log 02:31:51,832 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/leader331/metadata-v1-1.log 02:31:51,832 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader331: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:51.810Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8223461585593332524/leader331/snapshot-00000000685EAB57-30479E80.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:51,832 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - leader331: local snapshot saved in 4.856 ms, deleting Pekko-persisted snapshots 02:31:51,833 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - leader331: Recovery completed in 5.832 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:51,835 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader331: Set currentTerm=1, votedFor=null 02:31:51,835 PM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - leader331: Local TermInfo store seeded with TermInfo{term=1} 02:31:51,835 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader331: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6ef027dc 02:31:51,835 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader331: Moving last applied index from -1 to -1 02:31:51,835 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - leader331: journal recovery completedin 193.8 μs with journalIndex=1 02:31:51,835 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader331: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:52,169 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:31:52,169 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - leader331 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.8:17355, Removed) 02:31:52,169 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - leader331 (Follower): Received ElectionTimeout - switching to Candidate 02:31:52,170 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader331 (Candidate): Election: Candidate has following voting peers: [follower333, follower332] 02:31:52,173 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader331: Set currentTerm=2, votedFor=leader331 02:31:52,173 PM [test-pekko.actor.default-dispatcher-16] [INFO] Candidate - leader331 (Candidate): Starting new election term 2 02:31:52,173 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader331 (Candidate): Sending RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} to peer follower333 02:31:52,173 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader331 (Candidate): Sending RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} to peer follower332 02:31:52,173 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower332 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:52,173 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower332 (Follower): Cluster state: [] 02:31:52,173 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower332 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:52,173 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower333 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:52,173 PM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower332 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:52,174 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower333 (Follower): Cluster state: [] 02:31:52,174 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower333 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:52,174 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower333 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:52,174 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader331 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:31:52,176 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=2, votedFor=null 02:31:52,176 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower332 (Follower): In requestVote: RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:52,177 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=2, votedFor=null 02:31:52,177 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower333 (Follower): In requestVote: RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:52,179 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=2, votedFor=leader331 02:31:52,179 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower332 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:31:52,180 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader331 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:31:52,180 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader331 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:31:52,180 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Election: Leader has following peers: [follower333, follower332, follower334] 02:31:52,180 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:52,180 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:52,180 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:52,180 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader331 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:31:52,180 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower332: Last sync leader does not match current leader leader331, need to catch up to -1 02:31:52,180 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower332 updated: matchIndex: -1, nextIndex: 0 02:31:52,180 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:52,180 PM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower334 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term [WARN] [06/27/2025 14:31:52.180] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower332] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower332, initialSyncDone=false] 02:31:52,180 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=2, votedFor=leader331 02:31:52,180 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower333 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:31:52,181 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower333: Last sync leader does not match current leader leader331, need to catch up to -1 02:31:52,181 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower333 updated: matchIndex: -1, nextIndex: 0 02:31:52,181 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:52.181] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower333] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower333, initialSyncDone=false] 02:31:52,182 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower334: Set currentTerm=2, votedFor=null 02:31:52,182 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower334: Last sync leader does not match current leader leader331, need to catch up to -1 02:31:52,182 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower334 updated: matchIndex: -1, nextIndex: 0 02:31:52,183 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:52.183] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower334] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower334, initialSyncDone=false] 02:31:52,188 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:31:52,188 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 02:31:52,188 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 02:31:52,188 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader331: Persist data index=0 term=2 command=zero 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader331: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 02:31:52,188 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: received 1 entries in 361.5 ms 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower333: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower332 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower332: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:52,188 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower334 02:31:52,189 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower334: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 02:31:52,189 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 02:31:52,189 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): Number of entries to be appended = 1 02:31:52,189 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 02:31:52,189 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): Append entry to log zero 02:31:52,189 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower332: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 02:31:52,189 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): Log size is now 1 02:31:52,189 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower332, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:31:52,189 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached -1 (of expected -1) 02:31:52,189 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryStoreCompleter - follower332: awaiting more completions to resolve 1 deferred callback(s) 02:31:52,189 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: received 1 entries in 373.6 ms 02:31:52,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower334 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 02:31:52,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower334 (Follower): Number of entries to be appended = 1 02:31:52,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower334 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 02:31:52,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower334 (Follower): Append entry to log zero 02:31:52,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower334: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 02:31:52,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower334 (Follower): Log size is now 1 02:31:52,189 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower332 updated: matchIndex: 0, nextIndex: 1 02:31:52,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower334 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower334, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:31:52,189 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:52,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached -1 (of expected -1) 02:31:52,189 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryStoreCompleter - follower334: awaiting more completions to resolve 1 deferred callback(s) 02:31:52,189 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower334 updated: matchIndex: 0, nextIndex: 1 02:31:52,189 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: received 1 entries in 365.8 ms 02:31:52,189 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:52,195 PM [leader331-writer-152] [DEBUG] EntryStoreCompleter - leader331: 1 completion(s) pending 02:31:52,195 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: persisted 1 entries in 6.519 ms 02:31:52,195 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: waiting for more work 02:31:52,195 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryStoreCompleter - leader331: running 1 completion(s) 02:31:52,195 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader331: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} 02:31:52,195 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader331 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 02:31:52,195 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader331 (Leader): Setting last applied to 0 02:31:52,195 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader331: Moving last applied index from -1 to 0 02:31:52,195 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader331: Applying state for log index 0 data zero 02:31:52,195 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader331: applyState called: zero 02:31:52,195 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: received 1 entries in 318.6 μs 02:31:52,195 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: persisted 1 entries in 203.5 μs 02:31:52,196 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: waiting for more work 02:31:52,196 PM [follower332-writer-149] [DEBUG] EntryStoreCompleter - follower332: 1 completion(s) pending 02:31:52,196 PM [follower334-writer-151] [DEBUG] EntryStoreCompleter - follower334: 1 completion(s) pending 02:31:52,196 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryStoreCompleter - follower334: running 1 completion(s) 02:31:52,196 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: persisted 1 entries in 6.778 ms 02:31:52,196 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: waiting for more work 02:31:52,196 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: persisted 1 entries in 6.434 ms 02:31:52,196 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower334: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 02:31:52,196 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: waiting for more work 02:31:52,196 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryStoreCompleter - follower334: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c087fa30@7cf06c42 02:31:52,196 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryStoreCompleter - follower332: running 1 completion(s) 02:31:52,196 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower332: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 02:31:52,196 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryStoreCompleter - follower332: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c087fa30@5d3ef8d8 [WARN] [06/27/2025 14:31:52.196] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower334] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower334, initialSyncDone=true] [WARN] [06/27/2025 14:31:52.196] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower332] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower332, initialSyncDone=true] 02:31:52,300 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 02:31:52,300 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower333: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 02:31:52,300 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): Commit index set to 0 02:31:52,300 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower334 (Follower): Commit index set to 0 02:31:52,300 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 02:31:52,300 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 02:31:52,300 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 02:31:52,300 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower334 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 02:31:52,300 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower332 (Follower): Setting last applied to 0 02:31:52,300 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower334 (Follower): Setting last applied to 0 02:31:52,300 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower332: Moving last applied index from -1 to 0 02:31:52,300 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower334: Moving last applied index from -1 to 0 02:31:52,300 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower332: Applying state for log index 0 data zero 02:31:52,300 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower334: Applying state for log index 0 data zero 02:31:52,300 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower332: applyState called: zero 02:31:52,300 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower334: applyState called: zero 02:31:52,301 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: received 1 entries in 104.8 ms 02:31:52,301 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: persisted 1 entries in 239.0 μs 02:31:52,301 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: waiting for more work 02:31:52,301 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: received 1 entries in 105.3 ms 02:31:52,301 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: persisted 1 entries in 93.46 μs 02:31:52,301 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: waiting for more work 02:31:52,339 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 02:31:52,339 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 02:31:52,339 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 02:31:52,339 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader331: onRequestLeadership RequestLeadership [requestedFollowerId=follower333, replyTo=Actor[pekko://test/user/$c#1276727744]] 02:31:52,340 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader331: Initiating leader transfer 02:31:52,340 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader331: pauseLeader successfully completed - doing transfer 02:31:52,340 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Leader - leader331 (Leader): Attempting to transfer leadership 02:31:52,340 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower334: onLeaderTransitioning: LeaderTransitioning [leaderId=leader331] 02:31:52,340 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:52,340 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 02:31:52,340 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:52,340 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower332: onLeaderTransitioning: LeaderTransitioning [leaderId=leader331] 02:31:52,340 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:31:52,340 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 02:31:52,340 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 02:31:52,340 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower333: onLeaderTransitioning: LeaderTransitioning [leaderId=leader331] 02:31:52,419 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 02:31:52,419 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower333: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 02:31:52,540 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 02:31:52,540 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 02:31:52,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader331: leader transfer aborted 02:31:52,660 PM [test-pekko.actor.default-dispatcher-15] [WARN] RaftActorLeadershipTransferCohort - leader331: Failed to transfer leadership in 320.2 ms 02:31:52,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader331: LeadershipTransfer request from follower333 failed 02:31:52,660 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Leader - leader331 (Leader): Leadership transfer expired 02:31:52,660 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 02:31:52,660 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 02:31:52,692 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 02:31:52,692 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower332-notifier335#-1352977772] 02:31:52,703 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower332-collector336#821510728] 02:31:52,704 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower332] 02:31:52,704 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower332: terminating thread VirtualThread[#721,follower332-writer-149]/waiting 02:31:52,704 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: received 1 entries in 402.9 ms 02:31:52,704 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: persisted 1 entries in 40.95 μs 02:31:52,704 PM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@5d419f9d stopped 02:31:52,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower332: journal closed 02:31:52,705 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower332: journal closed 02:31:52,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower332: stopped executor with 0 remaining tasks 02:31:52,705 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower332: stopped executor with 0 remaining tasks 02:31:52,705 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower333-notifier337#1972391138] 02:31:52,705 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower333-collector338#1775302428] 02:31:52,705 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower333] 02:31:52,705 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower333: terminating thread VirtualThread[#722,follower333-writer-150]/waiting 02:31:52,706 PM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: received 1 entries in 885.0 ms 02:31:52,706 PM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: persisted 1 entries in 15.10 μs 02:31:52,706 PM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3f728ca1 stopped 02:31:52,706 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower333: journal closed 02:31:52,706 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower333: journal closed 02:31:52,706 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: stopped executor with 0 remaining tasks 02:31:52,706 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower333: stopped executor with 0 remaining tasks 02:31:52,706 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower334-notifier339#11539131] 02:31:52,706 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower334-collector340#-1390551965] 02:31:52,706 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower334] 02:31:52,707 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - follower334: terminating thread VirtualThread[#723,follower334-writer-151]/waiting 02:31:52,707 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: received 1 entries in 405.6 ms 02:31:52,707 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: persisted 1 entries in 15.95 μs 02:31:52,707 PM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@129123de stopped 02:31:52,707 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower334: journal closed 02:31:52,707 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower334: journal closed 02:31:52,707 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower334: stopped executor with 0 remaining tasks 02:31:52,707 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower334: stopped executor with 0 remaining tasks 02:31:52,707 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader331-notifier341#1242067500] 02:31:52,707 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader331-collector342#-741401068] 02:31:52,708 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader331] 02:31:52,708 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - leader331: terminating thread VirtualThread[#724,leader331-writer-152]/waiting 02:31:52,708 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: received 1 entries in 512.5 ms 02:31:52,708 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: persisted 1 entries in 24.71 μs 02:31:52,708 PM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@5308d7f3 stopped 02:31:52,708 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader331: journal closed 02:31:52,708 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - leader331: journal closed 02:31:52,708 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader331: stopped executor with 0 remaining tasks 02:31:52,708 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader331: stopped executor with 0 remaining tasks 02:31:52,708 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$c#1276727744] 02:31:52,712 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 02:31:52,712 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/27/2025 14:31:52.714] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower344]: org.opendaylight.controller.cluster.common.actor.Monitor@7d0991cd 02:31:52,715 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower344: Starting recovery with journal batch size 1000 02:31:52,715 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower344: started executor 02:31:52,715 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower344: started executor 02:31:52,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower344: metadata journal open: replayFrom=1 applyTo=0 02:31:52,716 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower344: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:52,716 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower344: journal open: applyTo=0 02:31:52,716 PM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@14ea2385 started 02:31:52,716 PM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: waiting for more work 02:31:52,717 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower344/TermInfo.properties does not exist 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower344/journal-v1-1.log 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower344/metadata-v1-1.log 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: no eligible files found 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower344: SnapshotOffer called. 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower344: initializing from snapshot taken at 2025-06-27T14:31:52.712Z 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=1, votedFor=null 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - follower344: Updated server config: isVoting: true, peers: [PeerInfo [id=leader343, address=pekko://test/user/leader343, votingState=VOTING], PeerInfo [id=follower346, address=pekko://test/user/follower346, votingState=NON_VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=VOTING]] 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for follower344 in 89.62 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower344: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:52.712Z 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower344: support() called 02:31:52,718 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower344/snapshot-00000000685EAB58-2A704200.tmp [WARN] [06/27/2025 14:31:52.719] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower345]: org.opendaylight.controller.cluster.common.actor.Monitor@2be487fd 02:31:52,720 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower345: Starting recovery with journal batch size 1000 02:31:52,720 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: started executor 02:31:52,720 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: started executor 02:31:52,721 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower345: metadata journal open: replayFrom=1 applyTo=0 02:31:52,721 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower345: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:52,721 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower345: journal open: applyTo=0 02:31:52,721 PM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@43d6b3d4 started 02:31:52,721 PM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: waiting for more work 02:31:52,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower344/snapshot-00000000685EAB58-2A704200.v1 02:31:52,722 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower345/TermInfo.properties does not exist 02:31:52,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower344/snapshot-00000000685EAB58-2A704200.v1 02:31:52,722 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower345/journal-v1-1.log 02:31:52,722 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower345/metadata-v1-1.log 02:31:52,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower344/journal-v1-1.log 02:31:52,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower344/metadata-v1-1.log 02:31:52,722 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: no eligible files found 02:31:52,722 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower345: SnapshotOffer called. 02:31:52,722 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower345: initializing from snapshot taken at 2025-06-27T14:31:52.712Z 02:31:52,722 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=1, votedFor=null 02:31:52,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:52.712Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower344/snapshot-00000000685EAB58-2A704200.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:52,722 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower344: local snapshot saved in 3.987 ms, deleting Pekko-persisted snapshots 02:31:52,722 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower345: Updated server config: isVoting: true, peers: [PeerInfo [id=leader343, address=pekko://test/user/leader343, votingState=VOTING], PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower346, address=pekko://test/user/follower346, votingState=NON_VOTING]] 02:31:52,723 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower344: Recovery completed in 4.964 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [06/27/2025 14:31:52.723] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower346]: org.opendaylight.controller.cluster.common.actor.Monitor@11e4b05d 02:31:52,723 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - Recovery snapshot applied for follower345 in 992.6 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:52,723 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower345: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:52.712Z 02:31:52,723 PM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower345: support() called 02:31:52,724 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower345/snapshot-00000000685EAB58-2A704200.tmp 02:31:52,724 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower346: Starting recovery with journal batch size 1000 02:31:52,724 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: started executor 02:31:52,725 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: started executor 02:31:52,725 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower346: metadata journal open: replayFrom=1 applyTo=0 02:31:52,726 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower346: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:52,726 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower346: journal open: applyTo=0 02:31:52,726 PM [follower346-writer-155] [DEBUG] JournalWriteTask - follower346: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3ae4318c started 02:31:52,726 PM [follower346-writer-155] [DEBUG] JournalWriteTask - follower346: waiting for more work 02:31:52,727 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower346/TermInfo.properties does not exist 02:31:52,727 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower346/journal-v1-1.log 02:31:52,727 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower346/metadata-v1-1.log 02:31:52,727 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: no eligible files found 02:31:52,727 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=1, votedFor=null 02:31:52,727 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower344: Local TermInfo store seeded with TermInfo{term=1} 02:31:52,727 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower344: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@482e6e2d 02:31:52,727 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower346: SnapshotOffer called. 02:31:52,727 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower344: Moving last applied index from -1 to -1 02:31:52,727 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower346: initializing from snapshot taken at 2025-06-27T14:31:52.712Z 02:31:52,727 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower346: Set currentTerm=1, votedFor=null 02:31:52,727 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - follower346: Updated server config: isVoting: false, peers: [PeerInfo [id=leader343, address=pekko://test/user/leader343, votingState=VOTING], PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=VOTING]] 02:31:52,727 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for follower346 in 113.6 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:52,728 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower344: journal recovery completedin 236.7 μs with journalIndex=1 02:31:52,728 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower346: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:52.712Z 02:31:52,728 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower346: support() called 02:31:52,728 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower344: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:52,728 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower345/snapshot-00000000685EAB58-2A704200.v1 [WARN] [06/27/2025 14:31:52.728] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader343]: org.opendaylight.controller.cluster.common.actor.Monitor@7a4cb40d 02:31:52,729 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActor - leader343: Starting recovery with journal batch size 1000 02:31:52,729 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: started executor 02:31:52,729 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower345/snapshot-00000000685EAB58-2A704200.v1 02:31:52,729 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: started executor 02:31:52,729 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower345/journal-v1-1.log 02:31:52,729 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower345/metadata-v1-1.log 02:31:52,729 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower345: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:52.712Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower345/snapshot-00000000685EAB58-2A704200.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:52,729 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower345: local snapshot saved in 5.517 ms, deleting Pekko-persisted snapshots 02:31:52,729 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader343: metadata journal open: replayFrom=1 applyTo=0 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader343: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - leader343: journal open: applyTo=0 02:31:52,730 PM [leader343-writer-156] [DEBUG] JournalWriteTask - leader343: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@189fa3c2 started 02:31:52,730 PM [leader343-writer-156] [DEBUG] JournalWriteTask - leader343: waiting for more work 02:31:52,730 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower345: Recovery completed in 7.696 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/leader343/TermInfo.properties does not exist 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/leader343/journal-v1-1.log 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/leader343/metadata-v1-1.log 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: no eligible files found 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - leader343: SnapshotOffer called. 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - leader343: initializing from snapshot taken at 2025-06-27T14:31:52.712Z 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=1, votedFor=null 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - leader343: Updated server config: isVoting: true, peers: [PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower346, address=pekko://test/user/follower346, votingState=NON_VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=VOTING]] 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for leader343 in 55.03 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader343: migrating from Pekko persistent-snapshot taken at 2025-06-27T14:31:52.712Z 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - leader343: support() called 02:31:52,730 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/leader343/snapshot-00000000685EAB58-2A704200.tmp 02:31:52,731 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower346/snapshot-00000000685EAB58-2A704200.tmp 02:31:52,732 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=1, votedFor=null 02:31:52,732 PM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower345: Local TermInfo store seeded with TermInfo{term=1} 02:31:52,732 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower345: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@29de6a29 02:31:52,732 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower345: Moving last applied index from -1 to -1 02:31:52,733 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower345: journal recovery completedin 188.9 μs with journalIndex=1 02:31:52,733 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower345: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:52,733 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/leader343/snapshot-00000000685EAB58-2A704200.v1 02:31:52,733 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/leader343/snapshot-00000000685EAB58-2A704200.v1 02:31:52,733 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower346/snapshot-00000000685EAB58-2A704200.v1 02:31:52,733 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/leader343/journal-v1-1.log 02:31:52,733 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/leader343/metadata-v1-1.log 02:31:52,733 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower346/snapshot-00000000685EAB58-2A704200.v1 02:31:52,733 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader343: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:52.712Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/leader343/snapshot-00000000685EAB58-2A704200.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:52,733 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower346/journal-v1-1.log 02:31:52,733 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower346/metadata-v1-1.log 02:31:52,733 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader343: local snapshot saved in 2.902 ms, deleting Pekko-persisted snapshots 02:31:52,733 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T14:31:52.712Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5971573928273122633/follower346/snapshot-00000000685EAB58-2A704200.v1, size=275, entryCompress=NONE, stateCompress=NONE} 02:31:52,733 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower346: local snapshot saved in 5.736 ms, deleting Pekko-persisted snapshots 02:31:52,734 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader343: Recovery completed in 3.703 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:52,734 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower346: Recovery completed in 6.805 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:31:52,736 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=1, votedFor=null 02:31:52,736 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader343: Local TermInfo store seeded with TermInfo{term=1} 02:31:52,736 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader343: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@1250af6c 02:31:52,737 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractReplicatedLog - leader343: Moving last applied index from -1 to -1 02:31:52,737 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] JournalRecovery - leader343: journal recovery completedin 223.5 μs with journalIndex=1 02:31:52,737 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower346: Set currentTerm=1, votedFor=null 02:31:52,737 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower346: Local TermInfo store seeded with TermInfo{term=1} 02:31:52,737 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower346: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@781d3ae 02:31:52,737 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader343: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:52,737 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower346: Moving last applied index from -1 to -1 02:31:52,738 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower346: journal recovery completedin 245.3 μs with journalIndex=1 02:31:52,738 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower346: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 02:31:53,80 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:31:53,80 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - leader343 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.8:17355, Removed) 02:31:53,80 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - leader343 (Follower): Received ElectionTimeout - switching to Candidate 02:31:53,80 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader343 (Candidate): Election: Candidate has following voting peers: [follower344, follower345] 02:31:53,84 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=2, votedFor=leader343 02:31:53,84 PM [test-pekko.actor.default-dispatcher-16] [INFO] Candidate - leader343 (Candidate): Starting new election term 2 02:31:53,84 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader343 (Candidate): Sending RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} to peer follower344 02:31:53,84 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader343 (Candidate): Sending RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} to peer follower345 02:31:53,84 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader343 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:31:53,84 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower344 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:53,84 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower345 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:53,84 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower344 (Follower): Cluster state: [] 02:31:53,84 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower344 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:53,84 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower345 (Follower): Cluster state: [] 02:31:53,84 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower344 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:53,84 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower345 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:53,84 PM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower345 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:31:53,87 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=2, votedFor=null 02:31:53,87 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=2, votedFor=null 02:31:53,87 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower344 (Follower): In requestVote: RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:53,87 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower345 (Follower): In requestVote: RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:53,93 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=2, votedFor=leader343 02:31:53,93 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower344 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:31:53,93 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader343 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:31:53,93 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader343 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:31:53,93 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Election: Leader has following peers: [follower344, follower346, follower345] 02:31:53,93 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,93 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,93 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower344: Last sync leader does not match current leader leader343, need to catch up to -1 02:31:53,93 PM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower346 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:31:53,94 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=2, votedFor=leader343 02:31:53,94 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower345 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} [WARN] [06/27/2025 14:31:53.094] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower344] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower344, initialSyncDone=false] 02:31:53,94 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,94 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader343 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:31:53,94 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower345: Last sync leader does not match current leader leader343, need to catch up to -1 [WARN] [06/27/2025 14:31:53.094] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower345] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower345, initialSyncDone=false] 02:31:53,95 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower344 updated: matchIndex: -1, nextIndex: 0 02:31:53,95 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,95 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower345 updated: matchIndex: -1, nextIndex: 0 02:31:53,95 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,96 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower346: Set currentTerm=2, votedFor=null 02:31:53,96 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower346: Last sync leader does not match current leader leader343, need to catch up to -1 02:31:53,97 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader343 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower346 updated: matchIndex: -1, nextIndex: 0 02:31:53,97 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:53.097] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower346] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower346, initialSyncDone=false] 02:31:53,140 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:31:53,140 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 02:31:53,141 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 02:31:53,141 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower344] 02:31:53,152 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower344: terminating thread VirtualThread[#725,follower344-writer-153]/waiting 02:31:53,152 PM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: received 1 entries in 435.2 ms 02:31:53,152 PM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: persisted 1 entries in 36.05 μs 02:31:53,152 PM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@14ea2385 stopped 02:31:53,152 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower344: journal closed 02:31:53,152 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower344: journal closed 02:31:53,152 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: stopped executor with 0 remaining tasks 02:31:53,152 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower344: stopped executor with 0 remaining tasks 02:31:53,152 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower346] 02:31:53,163 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower346: terminating thread VirtualThread[#727,follower346-writer-155]/waiting 02:31:53,163 PM [follower346-writer-155] [DEBUG] JournalWriteTask - follower346: received 1 entries in 436.9 ms 02:31:53,163 PM [follower346-writer-155] [DEBUG] JournalWriteTask - follower346: persisted 1 entries in 19.39 μs 02:31:53,163 PM [follower346-writer-155] [DEBUG] JournalWriteTask - follower346: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3ae4318c stopped 02:31:53,163 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower346: journal closed 02:31:53,163 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower346: journal closed 02:31:53,163 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: stopped executor with 0 remaining tasks 02:31:53,163 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower346: stopped executor with 0 remaining tasks 02:31:53,164 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 02:31:53,164 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 02:31:53,164 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader343: onRequestLeadership RequestLeadership [requestedFollowerId=follower345, replyTo=Actor[pekko://test/user/$d#-1458421918]] 02:31:53,164 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader343: Initiating leader transfer [WARN] [06/27/2025 14:31:53.164] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/leader343]: LeaderTransitioning [leaderId=leader343] [WARN] [06/27/2025 14:31:53.165] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/leader343]: LeaderTransitioning [leaderId=leader343] 02:31:53,165 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower345: onLeaderTransitioning: LeaderTransitioning [leaderId=leader343] 02:31:53,165 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader343: pauseLeader successfully completed - doing transfer 02:31:53,165 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Leader - leader343 (Leader): Attempting to transfer leadership 02:31:53,165 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:53.165] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:31:53,165 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:53.165] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:31:53,165 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,166 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower345: Lagging 0 entries behind leader leader343 and reached -1 (of expected -1) [WARN] [06/27/2025 14:31:53.166] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower345] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower345, initialSyncDone=true] 02:31:53,166 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Leader - leader343 (Leader): tryToCompleteLeadershipTransfer: followerId: follower345, matchIndex: -1, lastIndex: -1, isVoting: true 02:31:53,166 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Leader - leader343 (Leader): Follower's log matches - sending ElectionTimeout 02:31:53,166 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:53.166] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:31:53,166 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 14:31:53.166] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:31:53,166 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,166 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower345: Lagging 0 entries behind leader leader343 and reached -1 (of expected -1) 02:31:53,167 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:31:53,167 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower345 (Follower): Received TimeoutNow - switching to Candidate 02:31:53,167 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - follower345 (Candidate): Election: Candidate has following voting peers: [leader343, follower344] 02:31:53,167 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Leader - leader343 (Leader): Leader transfer complete 02:31:53,167 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader343: leader transfer complete - waiting for new leader 02:31:53,171 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=3, votedFor=follower345 02:31:53,171 PM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - follower345 (Candidate): Starting new election term 3 02:31:53,171 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - follower345 (Candidate): Sending RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} to peer leader343 02:31:53,171 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - follower345 (Candidate): Sending RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} to peer follower344 02:31:53,171 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - follower345 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 02:31:53,171 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader343 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:31:53,171 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader343 (Leader): Cluster state: [] 02:31:53,171 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader343 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:31:53,171 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractLeader - leader343 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower [WARN] [06/27/2025 14:31:53.171] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/follower345]: RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} 02:31:53,173 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=3, votedFor=null 02:31:53,174 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader343 (Leader): Leadership transfer in progress - processing RequestVote 02:31:53,174 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader343 (Leader): In requestVote: RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=3, votedFor=follower345 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader343 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader343 (Leader) :- Switching from behavior Leader to Follower, election term: 3 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - follower345 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - follower345 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower345 (Leader): Election: Leader has following peers: [leader343, follower344, follower346] 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower leader343: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower follower344: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower follower346: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - follower345 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - leader343: Last sync leader does not match current leader follower345, need to catch up to -1 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader343: leader changed to follower345 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorLeadershipTransferCohort - leader343: Successfully transferred leadership to follower345 in 12.52 ms 02:31:53,176 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader343: Leadership transferred successfully to follower345 [WARN] [06/27/2025 14:31:53.177] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/follower345]: AppendEntries{term=3, leaderId=follower345, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} [WARN] [06/27/2025 14:31:53.177] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/follower345]: AppendEntries{term=3, leaderId=follower345, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} [WARN] [06/27/2025 14:31:53.177] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/leader343] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader343, initialSyncDone=false] 02:31:53,177 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower345 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader343 updated: matchIndex: -1, nextIndex: 0 02:31:53,177 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower leader343: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:31:53,215 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 02:31:53,215 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower344-notifier347#2017381600] 02:31:53,226 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower344-collector348#-1568896367] 02:31:53,226 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower345-notifier349#1182437659] 02:31:53,227 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower345-collector350#1872222267] 02:31:53,227 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower345] 02:31:53,227 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower345: terminating thread VirtualThread[#726,follower345-writer-154]/waiting 02:31:53,227 PM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: received 1 entries in 506.0 ms 02:31:53,227 PM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: persisted 1 entries in 40.30 μs 02:31:53,227 PM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@43d6b3d4 stopped 02:31:53,228 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower345: journal closed 02:31:53,228 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower345: journal closed 02:31:53,228 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: stopped executor with 0 remaining tasks 02:31:53,228 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: stopped executor with 0 remaining tasks 02:31:53,228 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower346-notifier351#1951219835] 02:31:53,228 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower346-collector352#758845774] 02:31:53,228 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader343-notifier353#537137374] 02:31:53,228 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader343-collector354#518507481] 02:31:53,229 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader343] 02:31:53,229 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - leader343: terminating thread VirtualThread[#728,leader343-writer-156]/waiting 02:31:53,229 PM [leader343-writer-156] [DEBUG] JournalWriteTask - leader343: received 1 entries in 499.9 ms 02:31:53,230 PM [leader343-writer-156] [DEBUG] JournalWriteTask - leader343: persisted 1 entries in 20.11 μs 02:31:53,230 PM [leader343-writer-156] [DEBUG] JournalWriteTask - leader343: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@189fa3c2 stopped 02:31:53,230 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader343: journal closed 02:31:53,230 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - leader343: journal closed 02:31:53,230 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: stopped executor with 0 remaining tasks 02:31:53,230 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: stopped executor with 0 remaining tasks 02:31:53,230 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$d#-1458421918] 02:31:53,232 PM [test-pekko.actor.default-dispatcher-11] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [INFO] [06/27/2025 14:31:53.232] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [06/27/2025 14:31:53.233] [test-pekko.actor.default-dispatcher-16] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.170.8:17355) 02:31:53,235 PM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.8:17355] - Exiting completed 02:31:53,235 PM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.8:17355] - Shutting down... 02:31:53,235 PM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.8:17355] - Successfully shut down [INFO] [06/27/2025 14:31:53.234] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.8:17355] - Exiting completed [INFO] [06/27/2025 14:31:53.234] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.8:17355] - Shutting down... [INFO] [06/27/2025 14:31:53.234] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.8:17355] - Successfully shut down 02:31:53,236 PM [test-pekko.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. [INFO] [06/27/2025 14:31:53.236] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.170.8:17355/system/remoting-terminator] Shutting down remote daemon. [INFO] [06/27/2025 14:31:53.236] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.170.8:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 02:31:53,236 PM [test-pekko.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. 02:31:53,238 PM [test-pekko.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [INFO] [06/27/2025 14:31:53.238] [test-pekko.remote.default-remote-dispatcher-7] [pekko://test@10.30.170.8:17355/system/remoting-terminator] Remoting shut down. [DEBUG] [06/27/2025 14:31:53.239] [test-pekko.actor.internal-dispatcher-10] [EventStream] shutting down: StandardOutLogger