[DEBUG] [07/01/2025 21:09:21.438] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 09:09:21,440 PM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [07/01/2025 21:09:21.440] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [07/01/2025 21:09:21.443] [main] [EventStream(pekko://test)] Default Loggers started 09:09:21,480 PM [test-pekko.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.150:17355] with UID [-6099125203752379162] 09:09:21,480 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.150:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [07/01/2025 21:09:21.477] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.150:17355] with UID [-6099125203752379162] [INFO] [07/01/2025 21:09:21.478] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.150:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [07/01/2025 21:09:21.484] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.150:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [07/01/2025 21:09:21.484] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.150:17355] - Started up successfully [INFO] [07/01/2025 21:09:21.485] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.150:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [07/01/2025 21:09:21.485] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.150:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 09:09:21,486 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.150:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 09:09:21,486 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.150:17355] - Started up successfully 09:09:21,486 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.150:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 09:09:21,486 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.150:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 09:09:21,497 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 09:09:21,497 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [07/01/2025 21:09:21.502] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower279]: org.opendaylight.controller.cluster.common.actor.Monitor@6a831bf6 [INFO] [pekkoDeadLetter][07/01/2025 21:09:21.502] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower279] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:21,503 PM [test-pekko.actor.default-dispatcher-7] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower279] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:21,505 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - follower279: Starting recovery with journal batch size 1000 09:09:21,505 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: started executor 09:09:21,506 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: started executor 09:09:21,506 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower279: metadata journal open: replayFrom=1 applyTo=0 09:09:21,506 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower279: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:21,506 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - follower279: journal open: applyTo=0 09:09:21,507 PM [follower279-writer-128] [DEBUG] JournalWriteTask - follower279: journal writer started 09:09:21,507 PM [follower279-writer-128] [DEBUG] JournalWriteTask - follower279: waiting to receive commands 09:09:21,507 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower279/TermInfo.properties does not exist 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower279/journal-v1-1.log 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower279/metadata-v1-1.log 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: no eligible files found 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower279: SnapshotOffer called. 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower279: initializing from snapshot taken at 2025-07-01T21:09:21.497Z 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=1, votedFor=null 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - follower279: Updated server config: isVoting: true, peers: [PeerInfo [id=follower281, address=pekko://test/user/follower281, votingState=NON_VOTING], PeerInfo [id=leader278, address=pekko://test/user/leader278, votingState=VOTING], PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=VOTING]] 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - Recovery snapshot applied for follower279 in 91.99 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower279: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:21.497Z 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower279: support() called 09:09:21,508 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower279/snapshot-0000000068644E81-1D9F9E40.tmp 09:09:21,514 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower279/snapshot-0000000068644E81-1D9F9E40.v1 09:09:21,514 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower279/journal-v1-1.log 09:09:21,514 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower279/snapshot-0000000068644E81-1D9F9E40.v1 09:09:21,514 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower279/metadata-v1-1.log 09:09:21,514 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:21.497Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower279/snapshot-0000000068644E81-1D9F9E40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:21,515 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower279: local snapshot saved in 6.538 ms, deleting Pekko-persisted snapshots 09:09:21,515 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower279: Recovery completed in 7.632 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:21,523 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=1, votedFor=null 09:09:21,523 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower279: Local TermInfo store seeded with TermInfo{term=1} 09:09:21,523 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower279: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@12e54512 09:09:21,523 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower279: Moving last applied index from -1 to -1 09:09:21,523 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower279: empty journal: appending 0 entries 09:09:21,524 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower279: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:21,524 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower279: journal recovery completedin 445.1 μs with journalIndex=1 09:09:21,527 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower279: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:21,604 PM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower280] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [07/01/2025 21:09:21.604] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower280]: org.opendaylight.controller.cluster.common.actor.Monitor@7f2016a4 [INFO] [pekkoDeadLetter][07/01/2025 21:09:21.604] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower280] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:21,604 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - follower280: Starting recovery with journal batch size 1000 09:09:21,605 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: started executor 09:09:21,605 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: started executor 09:09:21,605 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower280: metadata journal open: replayFrom=1 applyTo=0 09:09:21,606 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower280: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:21,606 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - follower280: journal open: applyTo=0 09:09:21,606 PM [follower280-writer-129] [DEBUG] JournalWriteTask - follower280: journal writer started 09:09:21,606 PM [follower280-writer-129] [DEBUG] JournalWriteTask - follower280: waiting to receive commands 09:09:21,606 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower280/TermInfo.properties does not exist 09:09:21,606 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower280/journal-v1-1.log 09:09:21,606 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower280/metadata-v1-1.log 09:09:21,606 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: no eligible files found 09:09:21,606 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower280: SnapshotOffer called. 09:09:21,606 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower280: initializing from snapshot taken at 2025-07-01T21:09:21.497Z 09:09:21,607 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=1, votedFor=null 09:09:21,607 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - follower280: Updated server config: isVoting: true, peers: [PeerInfo [id=leader278, address=pekko://test/user/leader278, votingState=VOTING], PeerInfo [id=follower281, address=pekko://test/user/follower281, votingState=NON_VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 09:09:21,607 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - Recovery snapshot applied for follower280 in 73.72 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:21,607 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower280: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:21.497Z 09:09:21,607 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower280: support() called 09:09:21,607 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower280/snapshot-0000000068644E81-1D9F9E40.tmp [WARN] [07/01/2025 21:09:21.607] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower281]: org.opendaylight.controller.cluster.common.actor.Monitor@144522f4 [INFO] [pekkoDeadLetter][07/01/2025 21:09:21.607] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower281] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:21,608 PM [test-pekko.actor.default-dispatcher-7] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower281] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:21,608 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower281: Starting recovery with journal batch size 1000 09:09:21,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: started executor 09:09:21,608 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: started executor 09:09:21,609 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower281: metadata journal open: replayFrom=1 applyTo=0 09:09:21,609 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower281: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:21,610 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower281: journal open: applyTo=0 09:09:21,610 PM [follower281-writer-130] [DEBUG] JournalWriteTask - follower281: journal writer started 09:09:21,610 PM [follower281-writer-130] [DEBUG] JournalWriteTask - follower281: waiting to receive commands 09:09:21,610 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower280/snapshot-0000000068644E81-1D9F9E40.v1 09:09:21,610 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower281/TermInfo.properties does not exist 09:09:21,610 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower280/journal-v1-1.log 09:09:21,610 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower280/snapshot-0000000068644E81-1D9F9E40.v1 09:09:21,610 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower281/journal-v1-1.log 09:09:21,610 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower281/metadata-v1-1.log 09:09:21,610 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower280/metadata-v1-1.log 09:09:21,610 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: no eligible files found 09:09:21,610 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower281: SnapshotOffer called. 09:09:21,610 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower280: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:21.497Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower280/snapshot-0000000068644E81-1D9F9E40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:21,610 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower281: initializing from snapshot taken at 2025-07-01T21:09:21.497Z 09:09:21,611 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower280: local snapshot saved in 3.777 ms, deleting Pekko-persisted snapshots 09:09:21,611 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower281: Set currentTerm=1, votedFor=null 09:09:21,611 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PeerInfos - follower281: Updated server config: isVoting: false, peers: [PeerInfo [id=leader278, address=pekko://test/user/leader278, votingState=VOTING], PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 09:09:21,611 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - Recovery snapshot applied for follower281 in 91.78 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:21,611 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower281: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:21.497Z 09:09:21,611 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower281: support() called 09:09:21,611 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower281/snapshot-0000000068644E81-1D9F9E40.tmp 09:09:21,611 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower280: Recovery completed in 4.731 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:21,614 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=1, votedFor=null 09:09:21,614 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower280: Local TermInfo store seeded with TermInfo{term=1} 09:09:21,614 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower280: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@47395ad8 09:09:21,614 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower280: Moving last applied index from -1 to -1 09:09:21,614 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower280: empty journal: appending 0 entries 09:09:21,614 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower280: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:21,614 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower280: journal recovery completedin 285.2 μs with journalIndex=1 09:09:21,615 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower280: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:21,615 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower281/snapshot-0000000068644E81-1D9F9E40.v1 09:09:21,615 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower281/journal-v1-1.log 09:09:21,615 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower281/snapshot-0000000068644E81-1D9F9E40.v1 09:09:21,615 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower281/metadata-v1-1.log 09:09:21,615 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:21.497Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/follower281/snapshot-0000000068644E81-1D9F9E40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:21,615 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower281: local snapshot saved in 4.377 ms, deleting Pekko-persisted snapshots 09:09:21,616 PM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader278] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [07/01/2025 21:09:21.616] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader278]: org.opendaylight.controller.cluster.common.actor.Monitor@4e177320 [INFO] [pekkoDeadLetter][07/01/2025 21:09:21.616] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader278] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:21,616 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower281: Recovery completed in 5.436 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:21,616 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - leader278: Starting recovery with journal batch size 1000 09:09:21,617 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: started executor 09:09:21,617 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: started executor 09:09:21,617 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader278: metadata journal open: replayFrom=1 applyTo=0 09:09:21,618 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader278: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:21,618 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - leader278: journal open: applyTo=0 09:09:21,618 PM [leader278-writer-131] [DEBUG] JournalWriteTask - leader278: journal writer started 09:09:21,618 PM [leader278-writer-131] [DEBUG] JournalWriteTask - leader278: waiting to receive commands 09:09:21,618 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/leader278/TermInfo.properties does not exist 09:09:21,619 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/leader278/journal-v1-1.log 09:09:21,619 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/leader278/metadata-v1-1.log 09:09:21,619 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: no eligible files found 09:09:21,619 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - leader278: SnapshotOffer called. 09:09:21,619 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - leader278: initializing from snapshot taken at 2025-07-01T21:09:21.497Z 09:09:21,619 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader278: Set currentTerm=1, votedFor=null 09:09:21,619 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - leader278: Updated server config: isVoting: true, peers: [PeerInfo [id=follower281, address=pekko://test/user/follower281, votingState=NON_VOTING], PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 09:09:21,619 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower281: Set currentTerm=1, votedFor=null 09:09:21,619 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower281: Local TermInfo store seeded with TermInfo{term=1} 09:09:21,619 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower281: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@415b9026 09:09:21,619 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower281: Moving last applied index from -1 to -1 09:09:21,619 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower281: empty journal: appending 0 entries 09:09:21,619 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower281: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:21,619 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower281: journal recovery completedin 276.5 μs with journalIndex=1 09:09:21,620 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower281: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:21,620 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - Recovery snapshot applied for leader278 in 1.635 ms: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:21,620 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - leader278: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:21.497Z 09:09:21,620 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader278: support() called 09:09:21,621 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/leader278/snapshot-0000000068644E81-1D9F9E40.tmp 09:09:21,622 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/leader278/snapshot-0000000068644E81-1D9F9E40.v1 09:09:21,622 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/leader278/journal-v1-1.log 09:09:21,622 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/leader278/snapshot-0000000068644E81-1D9F9E40.v1 09:09:21,622 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/leader278/metadata-v1-1.log 09:09:21,622 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader278: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:21.497Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest12612254862729473939/leader278/snapshot-0000000068644E81-1D9F9E40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:21,622 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - leader278: local snapshot saved in 1.993 ms, deleting Pekko-persisted snapshots 09:09:21,623 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - leader278: Recovery completed in 4.701 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:21,625 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader278: Set currentTerm=1, votedFor=null 09:09:21,625 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - leader278: Local TermInfo store seeded with TermInfo{term=1} 09:09:21,625 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader278: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@20aecbc5 09:09:21,625 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader278: Moving last applied index from -1 to -1 09:09:21,625 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - leader278: empty journal: appending 0 entries 09:09:21,625 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader278: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:21,626 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - leader278: journal recovery completedin 277.2 μs with journalIndex=1 09:09:21,626 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader278: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:22,23 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:09:22,24 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - leader278 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.150:17355, Removed) 09:09:22,24 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - leader278 (Follower): Received ElectionTimeout - switching to Candidate 09:09:22,24 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader278 (Candidate): Election: Candidate has following voting peers: [follower280, follower279] 09:09:22,28 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader278: Set currentTerm=2, votedFor=leader278 09:09:22,28 PM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - leader278 (Candidate): Starting new election term 2 09:09:22,29 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader278 (Candidate): Sending RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} to peer follower280 09:09:22,29 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader278 (Candidate): Sending RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} to peer follower279 09:09:22,29 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - leader278 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:09:22,29 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower280 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:22,29 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower279 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:22,29 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower280 (Follower): Cluster state: [] 09:09:22,29 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower279 (Follower): Cluster state: [] 09:09:22,29 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower280 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:22,29 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower279 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:22,29 PM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - follower280 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:22,29 PM [test-pekko.actor.default-dispatcher-7] [INFO] Follower - follower279 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:22,33 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=2, votedFor=null 09:09:22,33 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower280 (Follower): In requestVote: RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:22,33 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=2, votedFor=null 09:09:22,33 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower279 (Follower): In requestVote: RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:22,35 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=2, votedFor=leader278 09:09:22,35 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower280 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:09:22,35 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader278 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:09:22,35 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader278 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:09:22,35 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Election: Leader has following peers: [follower281, follower280, follower279] 09:09:22,36 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=2, votedFor=leader278 09:09:22,36 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower279 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:09:22,36 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower281: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,36 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower280: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,37 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower279: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,37 PM [test-pekko.actor.default-dispatcher-7] [INFO] Follower - follower281 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader278, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 09:09:22,37 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader278 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:09:22,37 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower279: Last sync leader does not match current leader leader278, need to catch up to -1 [WARN] [07/01/2025 21:09:22.037] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower279] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower279, initialSyncDone=false] [INFO] [pekkoDeadLetter][07/01/2025 21:09:22.037] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower279] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:22,37 PM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:22,38 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower279 updated: matchIndex: -1, nextIndex: 0 09:09:22,38 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower280: Last sync leader does not match current leader leader278, need to catch up to -1 09:09:22,38 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower279: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,38 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower280 updated: matchIndex: -1, nextIndex: 0 09:09:22,38 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower280: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,38 PM [test-pekko.actor.default-dispatcher-13] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [07/01/2025 21:09:22.038] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower280] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower280, initialSyncDone=false] [INFO] [pekkoDeadLetter][07/01/2025 21:09:22.038] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower280] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:22,40 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower281: Set currentTerm=2, votedFor=null 09:09:22,40 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower281: Last sync leader does not match current leader leader278, need to catch up to -1 09:09:22,40 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader278 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower281 updated: matchIndex: -1, nextIndex: 0 09:09:22,40 PM [test-pekko.actor.default-dispatcher-7] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:22,40 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower281: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:22.040] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower281] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower281, initialSyncDone=false] [INFO] [pekkoDeadLetter][07/01/2025 21:09:22.040] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower281] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:22,80 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:09:22,80 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader278 starting 09:09:22,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader278: onShutDown 09:09:22,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader278: Initiating leader transfer 09:09:22,82 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower281: onLeaderTransitioning: LeaderTransitioning [leaderId=leader278] 09:09:22,82 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower279: onLeaderTransitioning: LeaderTransitioning [leaderId=leader278] 09:09:22,82 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower280: onLeaderTransitioning: LeaderTransitioning [leaderId=leader278] 09:09:22,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader278: pauseLeader successfully completed - doing transfer 09:09:22,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Leader - leader278 (Leader): Attempting to transfer leadership 09:09:22,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower281: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower280: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower279: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,82 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,82 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,82 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) [WARN] [07/01/2025 21:09:22.082] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower279] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower279, initialSyncDone=true] [WARN] [07/01/2025 21:09:22.083] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower281] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower281, initialSyncDone=true] 09:09:22,83 PM [test-pekko.actor.default-dispatcher-7] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][07/01/2025 21:09:22.082] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower281] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][07/01/2025 21:09:22.083] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower279] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:22,83 PM [test-pekko.actor.default-dispatcher-17] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [07/01/2025 21:09:22.083] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower280] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower280, initialSyncDone=true] [INFO] [pekkoDeadLetter][07/01/2025 21:09:22.083] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower280] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:22,83 PM [test-pekko.actor.default-dispatcher-16] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:09:22,154 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,155 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,155 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,274 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,274 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,274 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,396 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorLeadershipTransferCohort - leader278: leader transfer aborted 09:09:22,396 PM [test-pekko.actor.default-dispatcher-7] [WARN] RaftActorLeadershipTransferCohort - leader278: Failed to transfer leadership in 314.1 ms 09:09:22,396 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader278: leader transfer failed - sending PoisonPill 09:09:22,396 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Leader - leader278 (Leader): Leadership transfer expired 09:09:22,397 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRaftStorage - leader278: terminating thread VirtualThread[#713,leader278-writer-131]/waiting 09:09:22,397 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,397 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,397 PM [leader278-writer-131] [DEBUG] JournalWriteTask - leader278: received 1 commands after 778.6 ms 09:09:22,397 PM [leader278-writer-131] [DEBUG] JournalWriteTask - leader278: completed 1 commands in 30.16 μs 09:09:22,397 PM [leader278-writer-131] [DEBUG] JournalWriteTask - leader278: journal writer stopped 09:09:22,397 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:09:22,397 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - leader278: journal closed 09:09:22,397 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - leader278: journal closed 09:09:22,397 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader278: stopped executor with 0 remaining tasks 09:09:22,397 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader278: stopped executor with 0 remaining tasks [WARN] [07/01/2025 21:09:22.397] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader278] received dead letter from TestActor[pekko://test/user/follower281]: AppendEntriesReply{term=2, success=true, followerId=follower281, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} [WARN] [07/01/2025 21:09:22.398] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader278] received dead letter from TestActor[pekko://test/user/follower279]: AppendEntriesReply{term=2, success=true, followerId=follower279, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} [WARN] [07/01/2025 21:09:22.398] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader278] received dead letter from TestActor[pekko://test/user/follower280]: AppendEntriesReply{term=2, success=true, followerId=follower280, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:09:22,399 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader278 ending 09:09:22,401 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 09:09:22,401 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower279-notifier282#-1654786790] 09:09:22,412 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower279-collector283#-68378689] 09:09:22,412 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower279] 09:09:22,412 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower279: terminating thread VirtualThread[#708,follower279-writer-128]/waiting 09:09:22,413 PM [follower279-writer-128] [DEBUG] JournalWriteTask - follower279: received 1 commands after 906.0 ms 09:09:22,413 PM [follower279-writer-128] [DEBUG] JournalWriteTask - follower279: completed 1 commands in 19.19 μs 09:09:22,413 PM [follower279-writer-128] [DEBUG] JournalWriteTask - follower279: journal writer stopped 09:09:22,413 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower279: journal closed 09:09:22,413 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower279: journal closed 09:09:22,413 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower279: stopped executor with 0 remaining tasks 09:09:22,413 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower279: stopped executor with 0 remaining tasks 09:09:22,413 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower280-notifier284#-1437151750] 09:09:22,413 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower280-collector285#1378414004] 09:09:22,413 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower280] 09:09:22,413 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower280: terminating thread VirtualThread[#710,follower280-writer-129]/waiting 09:09:22,413 PM [follower280-writer-129] [DEBUG] JournalWriteTask - follower280: received 1 commands after 807.6 ms 09:09:22,413 PM [follower280-writer-129] [DEBUG] JournalWriteTask - follower280: completed 1 commands in 12.26 μs 09:09:22,413 PM [follower280-writer-129] [DEBUG] JournalWriteTask - follower280: journal writer stopped 09:09:22,414 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower280: journal closed 09:09:22,414 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower280: journal closed 09:09:22,414 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower280: stopped executor with 0 remaining tasks 09:09:22,414 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower280: stopped executor with 0 remaining tasks 09:09:22,414 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower281-notifier286#-530560098] 09:09:22,414 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower281-collector287#294354467] 09:09:22,414 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower281] 09:09:22,414 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower281: terminating thread VirtualThread[#712,follower281-writer-130]/waiting 09:09:22,414 PM [follower281-writer-130] [DEBUG] JournalWriteTask - follower281: received 1 commands after 804.3 ms 09:09:22,414 PM [follower281-writer-130] [DEBUG] JournalWriteTask - follower281: completed 1 commands in 10.52 μs 09:09:22,414 PM [follower281-writer-130] [DEBUG] JournalWriteTask - follower281: journal writer stopped 09:09:22,414 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower281: journal closed 09:09:22,414 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower281: journal closed 09:09:22,414 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower281: stopped executor with 0 remaining tasks 09:09:22,414 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower281: stopped executor with 0 remaining tasks 09:09:22,414 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader278-notifier288#-155093755] 09:09:22,415 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader278-collector289#168122960] 09:09:22,415 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader278] 09:09:22,417 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 09:09:22,417 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [07/01/2025 21:09:22.418] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower291]: org.opendaylight.controller.cluster.common.actor.Monitor@261ba762 09:09:22,419 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActor - follower291: Starting recovery with journal batch size 1000 09:09:22,419 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: started executor 09:09:22,420 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: started executor 09:09:22,420 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower291: metadata journal open: replayFrom=1 applyTo=0 09:09:22,421 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower291: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:22,421 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower291: journal open: applyTo=0 09:09:22,421 PM [follower291-writer-132] [DEBUG] JournalWriteTask - follower291: journal writer started 09:09:22,421 PM [follower291-writer-132] [DEBUG] JournalWriteTask - follower291: waiting to receive commands 09:09:22,422 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower291/TermInfo.properties does not exist 09:09:22,422 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower291/journal-v1-1.log 09:09:22,422 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower291/metadata-v1-1.log 09:09:22,422 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: no eligible files found 09:09:22,422 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower291: SnapshotOffer called. 09:09:22,422 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower291: initializing from snapshot taken at 2025-07-01T21:09:22.417Z 09:09:22,422 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=1, votedFor=null 09:09:22,422 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - follower291: Updated server config: isVoting: true, peers: [PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=VOTING], PeerInfo [id=follower293, address=pekko://test/user/follower293, votingState=NON_VOTING], PeerInfo [id=leader290, address=pekko://test/user/leader290, votingState=VOTING]] 09:09:22,422 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for follower291 in 93.98 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:22,423 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower291: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:22.417Z 09:09:22,423 PM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - follower291: support() called [WARN] [07/01/2025 21:09:22.423] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower292]: org.opendaylight.controller.cluster.common.actor.Monitor@37efced9 09:09:22,423 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower291/snapshot-0000000068644E82-18DAEA40.tmp 09:09:22,424 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower292: Starting recovery with journal batch size 1000 09:09:22,424 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: started executor 09:09:22,424 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: started executor 09:09:22,425 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower292: metadata journal open: replayFrom=1 applyTo=0 09:09:22,425 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower292: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:22,425 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower292: journal open: applyTo=0 09:09:22,425 PM [follower292-writer-133] [DEBUG] JournalWriteTask - follower292: journal writer started 09:09:22,425 PM [follower292-writer-133] [DEBUG] JournalWriteTask - follower292: waiting to receive commands 09:09:22,426 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower292/TermInfo.properties does not exist 09:09:22,426 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower292/journal-v1-1.log 09:09:22,426 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower292/metadata-v1-1.log 09:09:22,426 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: no eligible files found 09:09:22,426 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower292: SnapshotOffer called. 09:09:22,426 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower292: initializing from snapshot taken at 2025-07-01T21:09:22.417Z 09:09:22,426 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=1, votedFor=null 09:09:22,426 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - follower292: Updated server config: isVoting: true, peers: [PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=follower293, address=pekko://test/user/follower293, votingState=NON_VOTING], PeerInfo [id=leader290, address=pekko://test/user/leader290, votingState=VOTING]] 09:09:22,427 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for follower292 in 103.0 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:22,427 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower292: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:22.417Z 09:09:22,427 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower292: support() called 09:09:22,427 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower292/snapshot-0000000068644E82-18DAEA40.tmp [WARN] [07/01/2025 21:09:22.427] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower293]: org.opendaylight.controller.cluster.common.actor.Monitor@39282654 09:09:22,428 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActor - follower293: Starting recovery with journal batch size 1000 09:09:22,428 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: started executor 09:09:22,428 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: started executor 09:09:22,428 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower293: metadata journal open: replayFrom=1 applyTo=0 09:09:22,429 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower293: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:22,429 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - follower293: journal open: applyTo=0 09:09:22,429 PM [follower293-writer-134] [DEBUG] JournalWriteTask - follower293: journal writer started 09:09:22,429 PM [follower293-writer-134] [DEBUG] JournalWriteTask - follower293: waiting to receive commands 09:09:22,429 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower293/TermInfo.properties does not exist 09:09:22,429 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower293/journal-v1-1.log 09:09:22,429 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower293/metadata-v1-1.log 09:09:22,429 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: no eligible files found 09:09:22,429 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower293: SnapshotOffer called. 09:09:22,429 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower293: initializing from snapshot taken at 2025-07-01T21:09:22.417Z 09:09:22,429 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=1, votedFor=null 09:09:22,429 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PeerInfos - follower293: Updated server config: isVoting: false, peers: [PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=VOTING], PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=leader290, address=pekko://test/user/leader290, votingState=VOTING]] 09:09:22,430 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - Recovery snapshot applied for follower293 in 56.39 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:22,430 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower293: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:22.417Z 09:09:22,430 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower293: support() called 09:09:22,430 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower293/snapshot-0000000068644E82-18DAEA40.tmp [WARN] [07/01/2025 21:09:22.431] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader290]: org.opendaylight.controller.cluster.common.actor.Monitor@120f833 09:09:22,432 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader290: Starting recovery with journal batch size 1000 09:09:22,432 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: started executor 09:09:22,432 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: started executor 09:09:22,433 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader290: metadata journal open: replayFrom=1 applyTo=0 09:09:22,434 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader290: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:22,434 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - leader290: journal open: applyTo=0 09:09:22,434 PM [leader290-writer-135] [DEBUG] JournalWriteTask - leader290: journal writer started 09:09:22,434 PM [leader290-writer-135] [DEBUG] JournalWriteTask - leader290: waiting to receive commands 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/leader290/TermInfo.properties does not exist 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/leader290/journal-v1-1.log 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/leader290/metadata-v1-1.log 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: no eligible files found 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - leader290: SnapshotOffer called. 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - leader290: initializing from snapshot taken at 2025-07-01T21:09:22.417Z 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=1, votedFor=null 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - leader290: Updated server config: isVoting: true, peers: [PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=VOTING], PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=follower293, address=pekko://test/user/follower293, votingState=NON_VOTING]] 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for leader290 in 104.8 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader290: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:22.417Z 09:09:22,435 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader290: support() called 09:09:22,436 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/leader290/snapshot-0000000068644E82-18DAEA40.tmp 09:09:22,437 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower292/snapshot-0000000068644E82-18DAEA40.v1 09:09:22,437 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower291/snapshot-0000000068644E82-18DAEA40.v1 09:09:22,437 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower292/snapshot-0000000068644E82-18DAEA40.v1 09:09:22,438 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower292/journal-v1-1.log 09:09:22,438 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower292/metadata-v1-1.log 09:09:22,438 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower291/snapshot-0000000068644E82-18DAEA40.v1 09:09:22,438 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower293/snapshot-0000000068644E82-18DAEA40.v1 09:09:22,438 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower293/snapshot-0000000068644E82-18DAEA40.v1 09:09:22,438 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower293/journal-v1-1.log 09:09:22,438 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower291/journal-v1-1.log 09:09:22,438 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower293/metadata-v1-1.log 09:09:22,438 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower291/metadata-v1-1.log 09:09:22,438 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower293: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:22.417Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower293/snapshot-0000000068644E82-18DAEA40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:22,438 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower293: local snapshot saved in 8.460 ms, deleting Pekko-persisted snapshots 09:09:22,438 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:22.417Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower291/snapshot-0000000068644E82-18DAEA40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:22,438 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:22.417Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/follower292/snapshot-0000000068644E82-18DAEA40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:22,438 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower292: local snapshot saved in 11.60 ms, deleting Pekko-persisted snapshots 09:09:22,438 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower291: local snapshot saved in 15.74 ms, deleting Pekko-persisted snapshots 09:09:22,439 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower292: Recovery completed in 12.88 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:22,439 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower291: Recovery completed in 17.11 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:22,440 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower293: Recovery completed in 10.65 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:22,441 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/leader290/snapshot-0000000068644E82-18DAEA40.v1 09:09:22,442 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/leader290/snapshot-0000000068644E82-18DAEA40.v1 09:09:22,442 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/leader290/journal-v1-1.log 09:09:22,442 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/leader290/metadata-v1-1.log 09:09:22,442 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader290: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:22.417Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2382864978537353711/leader290/snapshot-0000000068644E82-18DAEA40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:22,442 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader290: local snapshot saved in 6.744 ms, deleting Pekko-persisted snapshots 09:09:22,443 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader290: Recovery completed in 7.882 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:22,444 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=1, votedFor=null 09:09:22,444 PM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower292: Local TermInfo store seeded with TermInfo{term=1} 09:09:22,444 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower292: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@3b960417 09:09:22,444 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower292: Moving last applied index from -1 to -1 09:09:22,444 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=1, votedFor=null 09:09:22,444 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower291: Local TermInfo store seeded with TermInfo{term=1} 09:09:22,444 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower291: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@4b8be0b4 09:09:22,444 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower291: Moving last applied index from -1 to -1 09:09:22,444 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - follower292: empty journal: appending 0 entries 09:09:22,445 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower292: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:22,445 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - follower292: journal recovery completedin 413.7 μs with journalIndex=1 09:09:22,445 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower291: empty journal: appending 0 entries 09:09:22,445 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower291: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:22,445 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower291: journal recovery completedin 485.1 μs with journalIndex=1 09:09:22,445 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower292: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:22,446 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower291: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:22,446 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=1, votedFor=null 09:09:22,446 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader290: Local TermInfo store seeded with TermInfo{term=1} 09:09:22,446 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader290: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@2b7cf45d 09:09:22,446 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=1, votedFor=null 09:09:22,446 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader290: Moving last applied index from -1 to -1 09:09:22,446 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower293: Local TermInfo store seeded with TermInfo{term=1} 09:09:22,446 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower293: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@73889ea 09:09:22,447 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower293: Moving last applied index from -1 to -1 09:09:22,447 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader290: empty journal: appending 0 entries 09:09:22,447 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower293: empty journal: appending 0 entries 09:09:22,447 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader290: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:22,447 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader290: journal recovery completedin 351.2 μs with journalIndex=1 09:09:22,447 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower293: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:22,447 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower293: journal recovery completedin 345.3 μs with journalIndex=1 09:09:22,447 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader290: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:22,447 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower293: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:22,843 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:09:22,844 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader290 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.150:17355, Removed) 09:09:22,844 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader290 (Follower): Received ElectionTimeout - switching to Candidate 09:09:22,844 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader290 (Candidate): Election: Candidate has following voting peers: [follower292, follower291] 09:09:22,862 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=2, votedFor=leader290 09:09:22,862 PM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - leader290 (Candidate): Starting new election term 2 09:09:22,862 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader290 (Candidate): Sending RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} to peer follower292 09:09:22,862 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader290 (Candidate): Sending RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} to peer follower291 09:09:22,863 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader290 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:09:22,863 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower292 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:22,863 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower291 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:22,863 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower291 (Follower): Cluster state: [] 09:09:22,863 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower291 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:22,863 PM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower291 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:22,863 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower292 (Follower): Cluster state: [] 09:09:22,863 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower292 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:22,863 PM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - follower292 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:22,865 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=2, votedFor=null 09:09:22,866 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower291 (Follower): In requestVote: RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:22,867 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=2, votedFor=null 09:09:22,867 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower292 (Follower): In requestVote: RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:22,868 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=2, votedFor=leader290 09:09:22,868 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower291 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:09:22,868 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader290 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:09:22,868 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader290 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:09:22,869 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader290 (Leader): Election: Leader has following peers: [follower292, follower291, follower293] 09:09:22,869 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,869 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,869 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower291: Last sync leader does not match current leader leader290, need to catch up to -1 [WARN] [07/01/2025 21:09:22.869] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=false] 09:09:22,869 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,869 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=2, votedFor=leader290 09:09:22,869 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower292 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:09:22,869 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower293 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader290, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 09:09:22,869 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower292: Last sync leader does not match current leader leader290, need to catch up to -1 [WARN] [07/01/2025 21:09:22.870] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower292] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower292, initialSyncDone=false] 09:09:22,870 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader290 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:09:22,870 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader290 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower291 updated: matchIndex: -1, nextIndex: 0 09:09:22,870 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,870 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader290 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower292 updated: matchIndex: -1, nextIndex: 0 09:09:22,870 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,871 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=2, votedFor=null 09:09:22,872 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower293: Last sync leader does not match current leader leader290, need to catch up to -1 [WARN] [07/01/2025 21:09:22.872] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower293] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower293, initialSyncDone=false] 09:09:22,872 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader290 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower293 updated: matchIndex: -1, nextIndex: 0 09:09:22,872 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,914 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:09:22,914 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 09:09:22,914 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 09:09:22,914 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 09:09:22,915 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader290: onRequestLeadership RequestLeadership [requestedFollowerId=follower292, replyTo=Actor[pekko://test/user/$a#-1628938120]] 09:09:22,916 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 09:09:22,916 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader290: Initiating leader transfer 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader290: pauseLeader successfully completed - doing transfer 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Leader - leader290 (Leader): Attempting to transfer leadership 09:09:22,917 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower292: onLeaderTransitioning: LeaderTransitioning [leaderId=leader290] 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,917 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower291: onLeaderTransitioning: LeaderTransitioning [leaderId=leader290] 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,917 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower291: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 09:09:22,917 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower293: onLeaderTransitioning: LeaderTransitioning [leaderId=leader290] 09:09:22,917 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower292: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) [WARN] [07/01/2025 21:09:22.917] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=true] 09:09:22,917 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower293: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Leader - leader290 (Leader): tryToCompleteLeadershipTransfer: followerId: follower292, matchIndex: -1, lastIndex: -1, isVoting: true 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Leader - leader290 (Leader): Follower's log matches - sending ElectionTimeout 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:22.917] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower292] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower292, initialSyncDone=true] 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,917 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower292: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) [WARN] [07/01/2025 21:09:22.917] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower293] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower293, initialSyncDone=true] 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Leader - leader290 (Leader): Leader transfer complete 09:09:22,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader290: leader transfer complete - waiting for new leader 09:09:22,917 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:09:22,917 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower293: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 09:09:22,917 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - follower292 (Follower): Received TimeoutNow - switching to Candidate 09:09:22,917 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - follower292 (Candidate): Election: Candidate has following voting peers: [follower291, leader290] 09:09:22,918 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower291: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 09:09:22,920 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=3, votedFor=follower292 09:09:22,920 PM [test-pekko.actor.default-dispatcher-18] [INFO] Candidate - follower292 (Candidate): Starting new election term 3 09:09:22,920 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - follower292 (Candidate): Sending RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} to peer follower291 09:09:22,920 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - follower292 (Candidate): Sending RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} to peer leader290 09:09:22,920 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - follower292 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 09:09:22,920 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower291 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:22,920 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower291 (Follower): Cluster state: [] 09:09:22,920 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower291 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:22,920 PM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower291 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 2 - updating term 09:09:22,921 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader290 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:22,921 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader290 (Leader): Cluster state: [] 09:09:22,921 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader290 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:22,921 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractLeader - leader290 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower 09:09:22,924 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=3, votedFor=null 09:09:22,924 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower291 (Follower): In requestVote: RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:22,925 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=3, votedFor=null 09:09:22,925 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader290 (Leader): Leadership transfer in progress - processing RequestVote 09:09:22,925 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader290 (Leader): In requestVote: RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:22,926 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=3, votedFor=follower292 09:09:22,926 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower291 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 09:09:22,926 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - follower292 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 09:09:22,926 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - follower292 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 09:09:22,926 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower292 (Leader): Election: Leader has following peers: [follower291, follower293, leader290] 09:09:22,926 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower291: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,926 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower293: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,926 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower leader290: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,927 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - follower292 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 09:09:22,927 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower293 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower292, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 09:09:22,927 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower291: Last sync leader does not match current leader follower292, need to catch up to -1 09:09:22,927 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower292 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower291 updated: matchIndex: -1, nextIndex: 0 09:09:22,927 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:22.927] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=false] 09:09:22,928 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=3, votedFor=follower292 09:09:22,928 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader290 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 09:09:22,928 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - leader290 (Leader) :- Switching from behavior Leader to Follower, election term: 3 09:09:22,928 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - leader290: Last sync leader does not match current leader follower292, need to catch up to -1 09:09:22,928 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorLeadershipTransferCohort - leader290: leader changed to follower292 09:09:22,928 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorLeadershipTransferCohort - leader290: Successfully transferred leadership to follower292 in 11.96 ms 09:09:22,928 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader290 updated: matchIndex: -1, nextIndex: 0 09:09:22,928 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader290: Leadership transferred successfully to follower292 09:09:22,928 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower leader290: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:22.929] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader290] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader290, initialSyncDone=false] 09:09:22,929 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=3, votedFor=null 09:09:22,929 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower293: Last sync leader does not match current leader follower292, need to catch up to -1 09:09:22,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower293 updated: matchIndex: -1, nextIndex: 0 [WARN] [07/01/2025 21:09:22.929] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower293] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower293, initialSyncDone=false] 09:09:22,929 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:22,967 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 09:09:22,967 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower291-notifier294#-1441307897] 09:09:22,979 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower291-collector295#131354118] 09:09:22,979 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower291] 09:09:22,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower291: terminating thread VirtualThread[#716,follower291-writer-132]/waiting 09:09:22,979 PM [follower291-writer-132] [DEBUG] JournalWriteTask - follower291: received 1 commands after 558.2 ms 09:09:22,979 PM [follower291-writer-132] [DEBUG] JournalWriteTask - follower291: completed 1 commands in 40.00 μs 09:09:22,979 PM [follower291-writer-132] [DEBUG] JournalWriteTask - follower291: journal writer stopped 09:09:22,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower291: journal closed 09:09:22,979 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower291: journal closed 09:09:22,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower291: stopped executor with 0 remaining tasks 09:09:22,979 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower291: stopped executor with 0 remaining tasks 09:09:22,980 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower292-notifier296#183961407] 09:09:22,980 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower292-collector297#-1372756514] 09:09:22,980 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower292] 09:09:22,980 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower292: terminating thread VirtualThread[#717,follower292-writer-133]/waiting 09:09:22,980 PM [follower292-writer-133] [DEBUG] JournalWriteTask - follower292: received 1 commands after 554.9 ms 09:09:22,980 PM [follower292-writer-133] [DEBUG] JournalWriteTask - follower292: completed 1 commands in 12.64 μs 09:09:22,980 PM [follower292-writer-133] [DEBUG] JournalWriteTask - follower292: journal writer stopped 09:09:22,980 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower292: journal closed 09:09:22,980 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower292: journal closed 09:09:22,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: stopped executor with 0 remaining tasks 09:09:22,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower292: stopped executor with 0 remaining tasks 09:09:22,981 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower293-notifier298#1060815110] 09:09:22,981 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower293-collector299#1152212196] 09:09:22,981 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower293] 09:09:22,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower293: terminating thread VirtualThread[#718,follower293-writer-134]/waiting 09:09:22,981 PM [follower293-writer-134] [DEBUG] JournalWriteTask - follower293: received 1 commands after 552.4 ms 09:09:22,981 PM [follower293-writer-134] [DEBUG] JournalWriteTask - follower293: completed 1 commands in 9.390 μs 09:09:22,981 PM [follower293-writer-134] [DEBUG] JournalWriteTask - follower293: journal writer stopped 09:09:22,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower293: journal closed 09:09:22,981 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower293: journal closed 09:09:22,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower293: stopped executor with 0 remaining tasks 09:09:22,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower293: stopped executor with 0 remaining tasks 09:09:22,982 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader290-notifier300#-507375472] 09:09:22,982 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader290-collector301#1560616312] 09:09:22,982 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader290] 09:09:22,982 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - leader290: terminating thread VirtualThread[#719,leader290-writer-135]/waiting 09:09:22,982 PM [leader290-writer-135] [DEBUG] JournalWriteTask - leader290: received 1 commands after 548.3 ms 09:09:22,982 PM [leader290-writer-135] [DEBUG] JournalWriteTask - leader290: completed 1 commands in 8.700 μs 09:09:22,982 PM [leader290-writer-135] [DEBUG] JournalWriteTask - leader290: journal writer stopped 09:09:22,982 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader290: journal closed 09:09:22,982 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - leader290: journal closed 09:09:22,982 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader290: stopped executor with 0 remaining tasks 09:09:22,982 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader290: stopped executor with 0 remaining tasks 09:09:22,982 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$a#-1628938120] 09:09:22,985 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 09:09:22,985 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [07/01/2025 21:09:22.987] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower303]: org.opendaylight.controller.cluster.common.actor.Monitor@6afef966 09:09:22,988 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower303: Starting recovery with journal batch size 1000 09:09:22,989 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: started executor 09:09:22,989 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: started executor 09:09:22,989 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower303: metadata journal open: replayFrom=1 applyTo=0 09:09:22,989 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower303: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:22,989 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower303: journal open: applyTo=0 09:09:22,990 PM [follower303-writer-136] [DEBUG] JournalWriteTask - follower303: journal writer started 09:09:22,990 PM [follower303-writer-136] [DEBUG] JournalWriteTask - follower303: waiting to receive commands 09:09:22,990 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower303/TermInfo.properties does not exist 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower303/journal-v1-1.log 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower303/metadata-v1-1.log 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: no eligible files found 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower303: SnapshotOffer called. 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower303: initializing from snapshot taken at 2025-07-01T21:09:22.985Z 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=1, votedFor=null 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - follower303: Updated server config: isVoting: true, peers: [PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=VOTING], PeerInfo [id=leader302, address=pekko://test/user/leader302, votingState=VOTING], PeerInfo [id=follower305, address=pekko://test/user/follower305, votingState=NON_VOTING]] 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for follower303 in 77.05 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower303: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:22.985Z 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower303: support() called [WARN] [07/01/2025 21:09:22.991] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower304]: org.opendaylight.controller.cluster.common.actor.Monitor@67ba99f9 09:09:22,991 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower303/snapshot-0000000068644E82-3AB5E840.tmp 09:09:22,992 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - follower304: Starting recovery with journal batch size 1000 09:09:22,992 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: started executor 09:09:22,992 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: started executor 09:09:22,993 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower304: metadata journal open: replayFrom=1 applyTo=0 09:09:22,993 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower304: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:22,993 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRaftStorage - follower304: journal open: applyTo=0 09:09:22,993 PM [follower304-writer-137] [DEBUG] JournalWriteTask - follower304: journal writer started 09:09:22,993 PM [follower304-writer-137] [DEBUG] JournalWriteTask - follower304: waiting to receive commands 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower304/TermInfo.properties does not exist 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower304/journal-v1-1.log 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower304/metadata-v1-1.log 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: no eligible files found 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PekkoRecovery - follower304: SnapshotOffer called. 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PekkoRecovery - follower304: initializing from snapshot taken at 2025-07-01T21:09:22.985Z 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=1, votedFor=null 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - follower304: Updated server config: isVoting: true, peers: [PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=leader302, address=pekko://test/user/leader302, votingState=VOTING], PeerInfo [id=follower305, address=pekko://test/user/follower305, votingState=NON_VOTING]] 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - Recovery snapshot applied for follower304 in 66.61 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower304: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:22.985Z 09:09:22,994 PM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower304: support() called 09:09:22,995 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower304/snapshot-0000000068644E82-3AB5E840.tmp 09:09:22,995 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower303/snapshot-0000000068644E82-3AB5E840.v1 09:09:22,995 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower303/snapshot-0000000068644E82-3AB5E840.v1 09:09:22,995 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower303/journal-v1-1.log 09:09:22,995 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower303/metadata-v1-1.log 09:09:22,995 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower303: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:22.985Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower303/snapshot-0000000068644E82-3AB5E840.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:22,995 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower303: local snapshot saved in 3.993 ms, deleting Pekko-persisted snapshots 09:09:22,996 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower303: Recovery completed in 4.946 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:22,997 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower305: Starting recovery with journal batch size 1000 09:09:22,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: started executor 09:09:22,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: started executor 09:09:22,997 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower305: metadata journal open: replayFrom=1 applyTo=0 [WARN] [07/01/2025 21:09:22.998] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower305]: org.opendaylight.controller.cluster.common.actor.Monitor@6f4f3bd4 09:09:22,998 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower305: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:22,998 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - follower305: journal open: applyTo=0 09:09:22,998 PM [follower305-writer-138] [DEBUG] JournalWriteTask - follower305: journal writer started 09:09:22,998 PM [follower305-writer-138] [DEBUG] JournalWriteTask - follower305: waiting to receive commands 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower305/TermInfo.properties does not exist 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower305/journal-v1-1.log 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower305/metadata-v1-1.log 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: no eligible files found 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower305: SnapshotOffer called. 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower305: initializing from snapshot taken at 2025-07-01T21:09:22.985Z 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower305: Set currentTerm=1, votedFor=null 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower305: Updated server config: isVoting: false, peers: [PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=VOTING], PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=leader302, address=pekko://test/user/leader302, votingState=VOTING]] 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for follower305 in 73.47 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower305: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:22.985Z 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower305: support() called 09:09:22,999 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower304/snapshot-0000000068644E82-3AB5E840.v1 09:09:22,999 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower304/snapshot-0000000068644E82-3AB5E840.v1 09:09:22,999 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower305/snapshot-0000000068644E82-3AB5E840.tmp [WARN] [07/01/2025 21:09:23.000] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader302]: org.opendaylight.controller.cluster.common.actor.Monitor@4b1669d1 09:09:23,0 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=1, votedFor=null 09:09:23,0 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower303: Local TermInfo store seeded with TermInfo{term=1} 09:09:23,0 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower303: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@607357a4 09:09:23,0 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower303: Moving last applied index from -1 to -1 09:09:23,0 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower303: empty journal: appending 0 entries 09:09:23,1 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower303: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:23,1 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActor - leader302: Starting recovery with journal batch size 1000 09:09:23,1 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower303: journal recovery completedin 626.9 μs with journalIndex=1 09:09:23,1 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: started executor 09:09:23,1 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: started executor 09:09:23,1 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader302: metadata journal open: replayFrom=1 applyTo=0 09:09:23,1 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower305/snapshot-0000000068644E82-3AB5E840.v1 09:09:23,1 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower305/snapshot-0000000068644E82-3AB5E840.v1 09:09:23,1 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower305/journal-v1-1.log 09:09:23,1 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower305/metadata-v1-1.log 09:09:23,1 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower303: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:23,1 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower305: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:22.985Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower305/snapshot-0000000068644E82-3AB5E840.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:23,1 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower305: local snapshot saved in 2.395 ms, deleting Pekko-persisted snapshots 09:09:23,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader302: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:23,2 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - leader302: journal open: applyTo=0 09:09:23,2 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower304/journal-v1-1.log 09:09:23,2 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower304/metadata-v1-1.log 09:09:23,2 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower304: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:22.985Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/follower304/snapshot-0000000068644E82-3AB5E840.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:23,2 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower304: local snapshot saved in 7.719 ms, deleting Pekko-persisted snapshots 09:09:23,2 PM [leader302-writer-139] [DEBUG] JournalWriteTask - leader302: journal writer started 09:09:23,2 PM [leader302-writer-139] [DEBUG] JournalWriteTask - leader302: waiting to receive commands 09:09:23,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/leader302/TermInfo.properties does not exist 09:09:23,2 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower305: Recovery completed in 3.675 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:23,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/leader302/journal-v1-1.log 09:09:23,2 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/leader302/metadata-v1-1.log 09:09:23,3 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: no eligible files found 09:09:23,3 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - leader302: SnapshotOffer called. 09:09:23,3 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - leader302: initializing from snapshot taken at 2025-07-01T21:09:22.985Z 09:09:23,3 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader302: Set currentTerm=1, votedFor=null 09:09:23,3 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - leader302: Updated server config: isVoting: true, peers: [PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=VOTING], PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=follower305, address=pekko://test/user/follower305, votingState=NON_VOTING]] 09:09:23,3 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for leader302 in 73.61 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:23,3 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader302: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:22.985Z 09:09:23,3 PM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - leader302: support() called 09:09:23,3 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/leader302/snapshot-0000000068644E82-3AB5E840.tmp 09:09:23,4 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower304: Recovery completed in 9.511 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:23,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower305: Set currentTerm=1, votedFor=null 09:09:23,5 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower305: Local TermInfo store seeded with TermInfo{term=1} 09:09:23,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower305: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@11bd5d8a 09:09:23,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower305: Moving last applied index from -1 to -1 09:09:23,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower305: empty journal: appending 0 entries 09:09:23,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower305: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:23,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower305: journal recovery completedin 231.0 μs with journalIndex=1 09:09:23,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower305: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:23,6 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/leader302/snapshot-0000000068644E82-3AB5E840.v1 09:09:23,6 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/leader302/snapshot-0000000068644E82-3AB5E840.v1 09:09:23,6 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/leader302/journal-v1-1.log 09:09:23,6 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/leader302/metadata-v1-1.log 09:09:23,6 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader302: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:22.985Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest892309486298799930/leader302/snapshot-0000000068644E82-3AB5E840.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:23,6 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader302: local snapshot saved in 3.030 ms, deleting Pekko-persisted snapshots 09:09:23,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=1, votedFor=null 09:09:23,6 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower304: Local TermInfo store seeded with TermInfo{term=1} 09:09:23,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower304: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@73f704ea 09:09:23,6 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader302: Recovery completed in 3.663 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:23,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower304: Moving last applied index from -1 to -1 09:09:23,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower304: empty journal: appending 0 entries 09:09:23,7 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower304: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:23,7 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower304: journal recovery completedin 233.2 μs with journalIndex=1 09:09:23,7 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower304: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:23,8 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader302: Set currentTerm=1, votedFor=null 09:09:23,8 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader302: Local TermInfo store seeded with TermInfo{term=1} 09:09:23,8 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader302: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@75bcc804 09:09:23,8 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractReplicatedLog - leader302: Moving last applied index from -1 to -1 09:09:23,8 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] JournalRecovery - leader302: empty journal: appending 0 entries 09:09:23,8 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader302: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:23,8 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] JournalRecovery - leader302: journal recovery completedin 248.8 μs with journalIndex=1 09:09:23,9 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader302: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:23,364 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:09:23,364 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - leader302 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.150:17355, Removed) 09:09:23,364 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - leader302 (Follower): Received ElectionTimeout - switching to Candidate 09:09:23,364 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader302 (Candidate): Election: Candidate has following voting peers: [follower304, follower303] 09:09:23,369 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader302: Set currentTerm=2, votedFor=leader302 09:09:23,369 PM [test-pekko.actor.default-dispatcher-18] [INFO] Candidate - leader302 (Candidate): Starting new election term 2 09:09:23,369 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader302 (Candidate): Sending RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} to peer follower304 09:09:23,369 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader302 (Candidate): Sending RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} to peer follower303 09:09:23,369 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader302 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:09:23,369 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower304 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:23,370 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower304 (Follower): Cluster state: [] 09:09:23,370 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower304 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:23,370 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower304 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:23,370 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower303 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:23,370 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower303 (Follower): Cluster state: [] 09:09:23,370 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower303 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:23,370 PM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower303 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:23,372 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=2, votedFor=null 09:09:23,373 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower304 (Follower): In requestVote: RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:23,374 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=2, votedFor=null 09:09:23,374 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower303 (Follower): In requestVote: RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:23,375 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=2, votedFor=leader302 09:09:23,375 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower304 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:09:23,375 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - leader302 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:09:23,375 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - leader302 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:09:23,375 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader302 (Leader): Election: Leader has following peers: [follower304, follower303, follower305] 09:09:23,375 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower304: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:23,375 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower303: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:23,375 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower305: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:23,375 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActorBehavior - leader302 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:09:23,375 PM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - follower305 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 09:09:23,376 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower304: Last sync leader does not match current leader leader302, need to catch up to -1 09:09:23,376 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader302 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower304 updated: matchIndex: -1, nextIndex: 0 09:09:23,376 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower304: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:23,376 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=2, votedFor=leader302 09:09:23,376 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower303 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} [WARN] [07/01/2025 21:09:23.376] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower304] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower304, initialSyncDone=false] 09:09:23,376 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower303: Last sync leader does not match current leader leader302, need to catch up to -1 09:09:23,376 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader302 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower303 updated: matchIndex: -1, nextIndex: 0 09:09:23,377 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower303: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:23.377] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower303] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower303, initialSyncDone=false] 09:09:23,379 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower305: Set currentTerm=2, votedFor=null 09:09:23,379 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower305: Last sync leader does not match current leader leader302, need to catch up to -1 09:09:23,379 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader302 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower305 updated: matchIndex: -1, nextIndex: 0 09:09:23,379 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower305: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:23.379] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower305] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower305, initialSyncDone=false] 09:09:23,413 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:09:23,413 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 09:09:23,414 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 09:09:23,414 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower304 starting 09:09:23,414 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower304: onShutDown 09:09:23,414 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower304: terminating thread VirtualThread[#721,follower304-writer-137]/waiting 09:09:23,414 PM [follower304-writer-137] [DEBUG] JournalWriteTask - follower304: received 1 commands after 420.8 ms 09:09:23,414 PM [follower304-writer-137] [DEBUG] JournalWriteTask - follower304: completed 1 commands in 45.16 μs 09:09:23,414 PM [follower304-writer-137] [DEBUG] JournalWriteTask - follower304: journal writer stopped 09:09:23,415 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower304: journal closed 09:09:23,415 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower304: journal closed 09:09:23,415 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower304: stopped executor with 0 remaining tasks 09:09:23,415 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower304: stopped executor with 0 remaining tasks 09:09:23,415 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower304 ending 09:09:23,415 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 09:09:23,415 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 09:09:23,415 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader302: onRequestLeadership RequestLeadership [requestedFollowerId=follower304, replyTo=Actor[pekko://test/user/$b#779002613]] 09:09:23,415 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader302: Initiating leader transfer 09:09:23,415 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader302: pauseLeader successfully completed - doing transfer [WARN] [07/01/2025 21:09:23.415] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: LeaderTransitioning [leaderId=leader302] 09:09:23,415 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Leader - leader302 (Leader): Attempting to transfer leadership 09:09:23,415 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower304: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:23,415 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower303: onLeaderTransitioning: LeaderTransitioning [leaderId=leader302] 09:09:23,415 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower305: onLeaderTransitioning: LeaderTransitioning [leaderId=leader302] 09:09:23,416 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower303: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:23.416] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:09:23,416 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower303: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 09:09:23,416 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower305: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:23.416] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower303] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower303, initialSyncDone=true] 09:09:23,416 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower305: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) [WARN] [07/01/2025 21:09:23.416] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower305] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower305, initialSyncDone=true] [WARN] [07/01/2025 21:09:23.495] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:09:23,614 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower303: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) [WARN] [07/01/2025 21:09:23.614] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:09:23,614 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower305: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 09:09:23,734 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader302: leader transfer aborted 09:09:23,734 PM [test-pekko.actor.default-dispatcher-16] [WARN] RaftActorLeadershipTransferCohort - leader302: Failed to transfer leadership in 318.5 ms 09:09:23,734 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader302: LeadershipTransfer request from follower304 failed 09:09:23,736 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Leader - leader302 (Leader): Leadership transfer expired 09:09:23,737 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower303: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 09:09:23,737 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower305: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) [WARN] [07/01/2025 21:09:23.736] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:09:23,769 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 09:09:23,769 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower303-notifier306#-1731263694] 09:09:23,780 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower303-collector307#166600809] 09:09:23,781 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower303] 09:09:23,781 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRaftStorage - follower303: terminating thread VirtualThread[#720,follower303-writer-136]/waiting 09:09:23,781 PM [follower303-writer-136] [DEBUG] JournalWriteTask - follower303: received 1 commands after 791.8 ms 09:09:23,781 PM [follower303-writer-136] [DEBUG] JournalWriteTask - follower303: completed 1 commands in 32.85 μs 09:09:23,781 PM [follower303-writer-136] [DEBUG] JournalWriteTask - follower303: journal writer stopped 09:09:23,782 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower303: journal closed 09:09:23,782 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - follower303: journal closed 09:09:23,782 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower303: stopped executor with 0 remaining tasks 09:09:23,782 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower303: stopped executor with 0 remaining tasks 09:09:23,782 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower304-notifier308#996994688] 09:09:23,782 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower304-collector309#1445304722] 09:09:23,783 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower304] 09:09:23,783 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower305-notifier310#-1442833182] 09:09:23,783 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower305-collector311#1749166838] 09:09:23,783 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower305] 09:09:23,784 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRaftStorage - follower305: terminating thread VirtualThread[#722,follower305-writer-138]/waiting 09:09:23,784 PM [follower305-writer-138] [DEBUG] JournalWriteTask - follower305: received 1 commands after 785.5 ms 09:09:23,784 PM [follower305-writer-138] [DEBUG] JournalWriteTask - follower305: completed 1 commands in 88.27 μs 09:09:23,784 PM [follower305-writer-138] [DEBUG] JournalWriteTask - follower305: journal writer stopped 09:09:23,784 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower305: journal closed 09:09:23,784 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - follower305: journal closed 09:09:23,784 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower305: stopped executor with 0 remaining tasks 09:09:23,784 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower305: stopped executor with 0 remaining tasks 09:09:23,784 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader302-notifier312#-1555975070] 09:09:23,784 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader302-collector313#-492032060] 09:09:23,785 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader302] 09:09:23,785 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRaftStorage - leader302: terminating thread VirtualThread[#723,leader302-writer-139]/waiting 09:09:23,785 PM [leader302-writer-139] [DEBUG] JournalWriteTask - leader302: received 1 commands after 782.7 ms 09:09:23,785 PM [leader302-writer-139] [DEBUG] JournalWriteTask - leader302: completed 1 commands in 16.89 μs 09:09:23,785 PM [leader302-writer-139] [DEBUG] JournalWriteTask - leader302: journal writer stopped 09:09:23,785 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - leader302: journal closed 09:09:23,785 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - leader302: journal closed 09:09:23,785 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader302: stopped executor with 0 remaining tasks 09:09:23,785 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader302: stopped executor with 0 remaining tasks 09:09:23,785 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$b#779002613] 09:09:23,789 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 09:09:23,789 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:09:23,791 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActor - follower315: Starting recovery with journal batch size 1000 09:09:23,791 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: started executor 09:09:23,792 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: started executor [WARN] [07/01/2025 21:09:23.792] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower315]: org.opendaylight.controller.cluster.common.actor.Monitor@314b90b3 09:09:23,792 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower315: metadata journal open: replayFrom=1 applyTo=0 09:09:23,792 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower315: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:23,792 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - follower315: journal open: applyTo=0 09:09:23,793 PM [follower315-writer-140] [DEBUG] JournalWriteTask - follower315: journal writer started 09:09:23,793 PM [follower315-writer-140] [DEBUG] JournalWriteTask - follower315: waiting to receive commands 09:09:23,793 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower315/TermInfo.properties does not exist 09:09:23,793 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower315/journal-v1-1.log 09:09:23,793 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower315/metadata-v1-1.log 09:09:23,793 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: no eligible files found 09:09:23,793 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower315: SnapshotOffer called. 09:09:23,793 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower315: initializing from snapshot taken at 2025-07-01T21:09:23.789Z 09:09:23,793 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=1, votedFor=null 09:09:23,793 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PeerInfos - follower315: Updated server config: isVoting: true, peers: [PeerInfo [id=follower317, address=pekko://test/user/follower317, votingState=NON_VOTING], PeerInfo [id=leader314, address=pekko://test/user/leader314, votingState=VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=VOTING]] 09:09:23,793 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - Recovery snapshot applied for follower315 in 74.85 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:23,794 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower315: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:23.789Z 09:09:23,794 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower315: support() called 09:09:23,794 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower315/snapshot-0000000068644E83-2F072F40.tmp [WARN] [07/01/2025 21:09:23.795] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower316]: org.opendaylight.controller.cluster.common.actor.Monitor@be9c441 09:09:23,796 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - follower316: Starting recovery with journal batch size 1000 09:09:23,797 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower316: started executor 09:09:23,797 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower316: started executor 09:09:23,797 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower316: metadata journal open: replayFrom=1 applyTo=0 09:09:23,798 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower315/snapshot-0000000068644E83-2F072F40.v1 09:09:23,798 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower316: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:23,798 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRaftStorage - follower316: journal open: applyTo=0 09:09:23,798 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower315/snapshot-0000000068644E83-2F072F40.v1 09:09:23,798 PM [follower316-writer-141] [DEBUG] JournalWriteTask - follower316: journal writer started 09:09:23,798 PM [follower316-writer-141] [DEBUG] JournalWriteTask - follower316: waiting to receive commands 09:09:23,798 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower315/journal-v1-1.log 09:09:23,798 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower315/metadata-v1-1.log 09:09:23,798 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:23.789Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower315/snapshot-0000000068644E83-2F072F40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:23,798 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower315: local snapshot saved in 4.806 ms, deleting Pekko-persisted snapshots 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower316/TermInfo.properties does not exist 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower316/journal-v1-1.log 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower316/metadata-v1-1.log 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower316: no eligible files found 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower316: SnapshotOffer called. 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower316: initializing from snapshot taken at 2025-07-01T21:09:23.789Z 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=1, votedFor=null 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PeerInfos - follower316: Updated server config: isVoting: true, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=follower317, address=pekko://test/user/follower317, votingState=NON_VOTING], PeerInfo [id=leader314, address=pekko://test/user/leader314, votingState=VOTING]] 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - Recovery snapshot applied for follower316 in 71.03 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower316: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:23.789Z 09:09:23,799 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower316: support() called 09:09:23,799 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower315: Recovery completed in 6.020 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:23,800 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower316: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower316/snapshot-0000000068644E83-2F072F40.tmp 09:09:23,801 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower316: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower316/snapshot-0000000068644E83-2F072F40.v1 09:09:23,801 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=1, votedFor=null 09:09:23,801 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower315: Local TermInfo store seeded with TermInfo{term=1} 09:09:23,801 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower315: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6390a4d 09:09:23,801 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower316: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower316/snapshot-0000000068644E83-2F072F40.v1 09:09:23,801 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower315: Moving last applied index from -1 to -1 09:09:23,801 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower316/journal-v1-1.log 09:09:23,801 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower316/metadata-v1-1.log 09:09:23,802 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower315: empty journal: appending 0 entries 09:09:23,802 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower316: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:23.789Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower316/snapshot-0000000068644E83-2F072F40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:23,802 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower316: local snapshot saved in 2.357 ms, deleting Pekko-persisted snapshots 09:09:23,802 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower315: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:23,802 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower315: journal recovery completedin 250.7 μs with journalIndex=1 09:09:23,802 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower316: Recovery completed in 3.050 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:23,802 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower315: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:23,804 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=1, votedFor=null 09:09:23,804 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower316: Local TermInfo store seeded with TermInfo{term=1} 09:09:23,804 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower316: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@65a9924f 09:09:23,804 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower316: Moving last applied index from -1 to -1 09:09:23,804 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower316: empty journal: appending 0 entries 09:09:23,805 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower316: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:23,805 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower316: journal recovery completedin 376.2 μs with journalIndex=1 09:09:23,805 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower316: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) [WARN] [07/01/2025 21:09:23.897] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower317]: org.opendaylight.controller.cluster.common.actor.Monitor@5b1e4282 09:09:23,898 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActor - follower317: Starting recovery with journal batch size 1000 09:09:23,899 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: started executor 09:09:23,899 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: started executor 09:09:23,899 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower317: metadata journal open: replayFrom=1 applyTo=0 09:09:23,900 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower317: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:23,900 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - follower317: journal open: applyTo=0 09:09:23,900 PM [follower317-writer-142] [DEBUG] JournalWriteTask - follower317: journal writer started 09:09:23,900 PM [follower317-writer-142] [DEBUG] JournalWriteTask - follower317: waiting to receive commands 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower317/TermInfo.properties does not exist 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower317/journal-v1-1.log 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower317/metadata-v1-1.log 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: no eligible files found 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower317: SnapshotOffer called. 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower317: initializing from snapshot taken at 2025-07-01T21:09:23.789Z 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=1, votedFor=null 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PeerInfos - follower317: Updated server config: isVoting: false, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=leader314, address=pekko://test/user/leader314, votingState=VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=VOTING]] 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - Recovery snapshot applied for follower317 in 77.67 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower317: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:23.789Z 09:09:23,901 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower317: support() called 09:09:23,902 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower317/snapshot-0000000068644E83-2F072F40.tmp [WARN] [07/01/2025 21:09:23.902] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader314]: org.opendaylight.controller.cluster.common.actor.Monitor@60ab9cb5 09:09:23,903 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader314: Starting recovery with journal batch size 1000 09:09:23,903 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader314: started executor 09:09:23,903 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader314: started executor 09:09:23,904 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader314: metadata journal open: replayFrom=1 applyTo=0 09:09:23,904 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower317/snapshot-0000000068644E83-2F072F40.v1 09:09:23,904 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower317/snapshot-0000000068644E83-2F072F40.v1 09:09:23,904 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader314: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:23,904 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRaftStorage - leader314: journal open: applyTo=0 09:09:23,904 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower317/journal-v1-1.log 09:09:23,904 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower317/metadata-v1-1.log 09:09:23,904 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: journal writer started 09:09:23,904 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: waiting to receive commands 09:09:23,904 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower317: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:23.789Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/follower317/snapshot-0000000068644E83-2F072F40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:23,904 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower317: local snapshot saved in 3.054 ms, deleting Pekko-persisted snapshots 09:09:23,905 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower317: Recovery completed in 3.927 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:23,905 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/leader314/TermInfo.properties does not exist 09:09:23,905 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/leader314/journal-v1-1.log 09:09:23,905 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/leader314/metadata-v1-1.log 09:09:23,905 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader314: no eligible files found 09:09:23,905 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - leader314: SnapshotOffer called. 09:09:23,905 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - leader314: initializing from snapshot taken at 2025-07-01T21:09:23.789Z 09:09:23,905 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=1, votedFor=null 09:09:23,905 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - leader314: Updated server config: isVoting: true, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=follower317, address=pekko://test/user/follower317, votingState=NON_VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=VOTING]] 09:09:23,905 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for leader314 in 58.52 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:23,906 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader314: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:23.789Z 09:09:23,906 PM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - leader314: support() called 09:09:23,906 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader314: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/leader314/snapshot-0000000068644E83-2F072F40.tmp 09:09:23,907 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=1, votedFor=null 09:09:23,908 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower317: Local TermInfo store seeded with TermInfo{term=1} 09:09:23,908 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower317: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@25720057 09:09:23,908 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower317: Moving last applied index from -1 to -1 09:09:23,908 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower317: empty journal: appending 0 entries 09:09:23,908 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower317: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:23,908 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower317: journal recovery completedin 296.5 μs with journalIndex=1 09:09:23,908 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader314: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/leader314/snapshot-0000000068644E83-2F072F40.v1 09:09:23,908 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader314: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/leader314/snapshot-0000000068644E83-2F072F40.v1 09:09:23,908 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower317: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:23,908 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/leader314/journal-v1-1.log 09:09:23,908 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/leader314/metadata-v1-1.log 09:09:23,908 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader314: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:23.789Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17270014040744234126/leader314/snapshot-0000000068644E83-2F072F40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:23,908 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader314: local snapshot saved in 2.964 ms, deleting Pekko-persisted snapshots 09:09:23,909 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - leader314: Recovery completed in 3.558 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:23,911 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=1, votedFor=null 09:09:23,911 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - leader314: Local TermInfo store seeded with TermInfo{term=1} 09:09:23,911 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader314: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@7e1487d 09:09:23,911 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader314: Moving last applied index from -1 to -1 09:09:23,911 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - leader314: empty journal: appending 0 entries 09:09:23,911 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - leader314: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:23,911 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - leader314: journal recovery completedin 271.8 μs with journalIndex=1 09:09:23,912 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader314: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:24,284 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:09:24,284 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - leader314 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.150:17355, Removed) 09:09:24,284 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - leader314 (Follower): Received ElectionTimeout - switching to Candidate 09:09:24,284 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader314 (Candidate): Election: Candidate has following voting peers: [follower315, follower316] 09:09:24,289 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=2, votedFor=leader314 09:09:24,289 PM [test-pekko.actor.default-dispatcher-18] [INFO] Candidate - leader314 (Candidate): Starting new election term 2 09:09:24,289 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader314 (Candidate): Sending RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} to peer follower315 09:09:24,289 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader314 (Candidate): Sending RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} to peer follower316 09:09:24,289 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader314 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:09:24,290 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower315 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:24,290 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower315 (Follower): Cluster state: [] 09:09:24,290 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower315 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:24,290 PM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower315 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:24,290 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower316 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:24,290 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower316 (Follower): Cluster state: [] 09:09:24,290 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower316 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:24,290 PM [test-pekko.actor.default-dispatcher-7] [INFO] Follower - follower316 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:24,292 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=2, votedFor=null 09:09:24,293 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower315 (Follower): In requestVote: RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:24,294 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=2, votedFor=null 09:09:24,294 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower316 (Follower): In requestVote: RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:24,295 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=2, votedFor=leader314 09:09:24,296 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower315 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:09:24,296 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader314 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:09:24,296 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader314 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:09:24,296 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader314 (Leader): Election: Leader has following peers: [follower315, follower317, follower316] 09:09:24,296 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:24,296 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:24,296 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:24,296 PM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader314 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:09:24,296 PM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower317 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 09:09:24,296 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower315 updated: matchIndex: -1, nextIndex: 0 09:09:24,296 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:24,296 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower315: Last sync leader does not match current leader leader314, need to catch up to -1 09:09:24,296 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=2, votedFor=leader314 09:09:24,296 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower316 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} [WARN] [07/01/2025 21:09:24.297] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower315] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower315, initialSyncDone=false] 09:09:24,297 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower316: Last sync leader does not match current leader leader314, need to catch up to -1 09:09:24,297 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower316 updated: matchIndex: -1, nextIndex: 0 09:09:24,297 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:24.297] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=false] 09:09:24,298 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=2, votedFor=null 09:09:24,298 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower317: Last sync leader does not match current leader leader314, need to catch up to -1 09:09:24,298 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower317 updated: matchIndex: -1, nextIndex: 0 09:09:24,298 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:24.298] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=false] 09:09:24,318 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:09:24,318 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 09:09:24,318 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader314: Persist data index=0 term=2 command=zero 09:09:24,318 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader314: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:09:24,318 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 09:09:24,318 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,318 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower315 09:09:24,318 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: received 1 commands after 413.5 ms 09:09:24,318 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower315: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:09:24,318 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - follower315 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:09:24,318 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - follower315 (Follower): Number of entries to be appended = 1 09:09:24,318 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - follower315 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 09:09:24,318 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - follower315 (Follower): Append entry to log zero 09:09:24,318 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] ReplicatedLogImpl - follower315: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:09:24,318 PM [follower315-writer-140] [DEBUG] JournalWriteTask - follower315: received 1 commands after 525.7 ms 09:09:24,319 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,319 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower317 09:09:24,319 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - follower315 (Follower): Log size is now 1 09:09:24,319 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - follower315 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower315, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:09:24,319 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached -1 (of expected -1) 09:09:24,319 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower317: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} [WARN] [07/01/2025 21:09:24.319] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower315] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower315, initialSyncDone=true] 09:09:24,319 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower317 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:09:24,319 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower317 (Follower): Number of entries to be appended = 1 09:09:24,319 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower317 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 09:09:24,319 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower317 (Follower): Append entry to log zero 09:09:24,319 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower317: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:09:24,319 PM [follower317-writer-142] [DEBUG] JournalWriteTask - follower317: received 1 commands after 419.1 ms 09:09:24,320 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,320 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 09:09:24,320 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:09:24,320 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader314: awaiting more completions to resolve 1 synchronized callback(s) 09:09:24,320 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower317 (Follower): Log size is now 1 09:09:24,320 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower317 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower317, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:09:24,320 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached -1 (of expected -1) [WARN] [07/01/2025 21:09:24.320] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=true] 09:09:24,323 PM [leader314-writer-143] [DEBUG] RaftStorageCompleter - leader314: 1 completion(s) pending 09:09:24,323 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader314: running 1 completion(s) 09:09:24,323 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: completed 1 commands in 5.413 ms 09:09:24,323 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: waiting to receive commands 09:09:24,323 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - leader314: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader314: completed synchronized callback AppendSubmittedCallback{entry=JournaledLogEntry{index=0, term=2, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.RaftActor$$Lambda/0x00000007c0879d90@2e6e6f8a} 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower315 updated: matchIndex: 0, nextIndex: 1 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader314 (Leader): Setting last applied to 0 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader314: Moving last applied index from -1 to 0 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader314: Applying state for log index 0 data zero 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - leader314: applyState called: zero 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,324 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: received 1 commands after 460.9 μs 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower317 updated: matchIndex: 0, nextIndex: 1 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,324 PM [leader314-writer-143] [DEBUG] EntryJournalV1 - leader314: updated meta to JournalMeta[replayFrom=1, applyTo=1] 09:09:24,324 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): Commit index set to 0 09:09:24,324 PM [leader314-writer-143] [DEBUG] RaftStorageCompleter - leader314: 1 completion(s) pending 09:09:24,324 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:09:24,324 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 09:09:24,324 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - leader314: running 1 completion(s) 09:09:24,324 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: completed 1 commands in 199.8 μs 09:09:24,324 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower315 (Follower): Setting last applied to 0 09:09:24,324 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: waiting to receive commands 09:09:24,324 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower317 (Follower): Commit index set to 0 09:09:24,324 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower315: Moving last applied index from -1 to 0 09:09:24,324 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:09:24,324 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower315: Applying state for log index 0 data zero 09:09:24,324 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower315: applyState called: zero 09:09:24,324 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower317 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 09:09:24,324 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower317 (Follower): Setting last applied to 0 09:09:24,324 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower317: Moving last applied index from -1 to 0 09:09:24,324 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower317: Applying state for log index 0 data zero 09:09:24,324 PM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower317: applyState called: zero 09:09:24,325 PM [follower315-writer-140] [DEBUG] RaftStorageCompleter - follower315: 1 completion(s) pending 09:09:24,325 PM [follower317-writer-142] [DEBUG] RaftStorageCompleter - follower317: 1 completion(s) pending 09:09:24,325 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower315: running 1 completion(s) 09:09:24,325 PM [follower315-writer-140] [DEBUG] JournalWriteTask - follower315: completed 1 commands in 6.337 ms 09:09:24,325 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower317: running 1 completion(s) 09:09:24,325 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower315: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:09:24,325 PM [follower315-writer-140] [DEBUG] JournalWriteTask - follower315: received 1 commands after 5.900 μs 09:09:24,325 PM [follower317-writer-142] [DEBUG] JournalWriteTask - follower317: completed 1 commands in 5.400 ms 09:09:24,325 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower317: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:09:24,325 PM [follower317-writer-142] [DEBUG] JournalWriteTask - follower317: received 1 commands after 7.900 μs 09:09:24,325 PM [follower315-writer-140] [DEBUG] EntryJournalV1 - follower315: updated meta to JournalMeta[replayFrom=1, applyTo=1] 09:09:24,325 PM [follower315-writer-140] [DEBUG] RaftStorageCompleter - follower315: 1 completion(s) pending 09:09:24,325 PM [follower315-writer-140] [DEBUG] JournalWriteTask - follower315: completed 1 commands in 161.2 μs 09:09:24,325 PM [follower315-writer-140] [DEBUG] JournalWriteTask - follower315: waiting to receive commands 09:09:24,325 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower315: running 1 completion(s) 09:09:24,325 PM [follower317-writer-142] [DEBUG] EntryJournalV1 - follower317: updated meta to JournalMeta[replayFrom=1, applyTo=1] 09:09:24,325 PM [follower317-writer-142] [DEBUG] RaftStorageCompleter - follower317: 1 completion(s) pending 09:09:24,325 PM [follower317-writer-142] [DEBUG] JournalWriteTask - follower317: completed 1 commands in 213.0 μs 09:09:24,325 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower317: running 1 completion(s) 09:09:24,325 PM [follower317-writer-142] [DEBUG] JournalWriteTask - follower317: waiting to receive commands 09:09:24,369 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 09:09:24,369 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 09:09:24,369 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader314: onShutDown 09:09:24,369 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader314: Initiating leader transfer 09:09:24,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower315: onLeaderTransitioning: LeaderTransitioning [leaderId=leader314] 09:09:24,369 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader314: pauseLeader successfully completed - doing transfer 09:09:24,369 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Leader - leader314 (Leader): Attempting to transfer leadership 09:09:24,369 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower317: onLeaderTransitioning: LeaderTransitioning [leaderId=leader314] 09:09:24,369 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,369 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,369 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,369 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 09:09:24,369 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:09:24,369 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:09:24,369 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 09:09:24,370 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Leader - leader314 (Leader): tryToCompleteLeadershipTransfer: followerId: follower315, matchIndex: 0, lastIndex: 0, isVoting: true 09:09:24,370 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Leader - leader314 (Leader): Follower's log matches - sending ElectionTimeout 09:09:24,370 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,370 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,370 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:09:24,370 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:09:24,370 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,370 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 09:09:24,370 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Leader - leader314 (Leader): Leader transfer complete 09:09:24,370 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader314: leader transfer complete - waiting for new leader 09:09:24,370 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:09:24,370 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower315 (Follower): Received TimeoutNow - switching to Candidate 09:09:24,370 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - follower315 (Candidate): Election: Candidate has following voting peers: [leader314, follower316] 09:09:24,379 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=3, votedFor=follower315 09:09:24,379 PM [test-pekko.actor.default-dispatcher-7] [INFO] Candidate - follower315 (Candidate): Starting new election term 3 09:09:24,380 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - follower315 (Candidate): Sending RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} to peer leader314 09:09:24,380 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - follower315 (Candidate): Sending RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} to peer follower316 09:09:24,380 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActorBehavior - follower315 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 09:09:24,380 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower316 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:24,380 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower316 (Follower): Cluster state: [] 09:09:24,380 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower316 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 09:09:24,380 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower316 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2}" message is greater than follower's term 2 - updating term 09:09:24,380 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader314 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:24,380 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader314 (Leader): Cluster state: [] 09:09:24,380 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader314 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 09:09:24,380 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractLeader - leader314 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2}" message is greater than leader's term 2 - switching to Follower 09:09:24,382 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=3, votedFor=null 09:09:24,382 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower316 (Follower): In requestVote: RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:24,384 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=3, votedFor=null 09:09:24,384 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader314 (Leader): Leadership transfer in progress - processing RequestVote 09:09:24,384 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader314 (Leader): In requestVote: RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 09:09:24,386 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=3, votedFor=follower315 09:09:24,386 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower316 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 09:09:24,386 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - follower315 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 09:09:24,386 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - follower315 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 09:09:24,386 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower315 (Leader): Election: Leader has following peers: [follower317, leader314, follower316] 09:09:24,386 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower317: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,386 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower leader314: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,386 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower316: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,386 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - follower315 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 09:09:24,386 PM [test-pekko.actor.default-dispatcher-7] [INFO] Follower - follower317 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower315, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 09:09:24,387 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=3, votedFor=follower315 09:09:24,387 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader314 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 09:09:24,387 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - leader314 (Leader) :- Switching from behavior Leader to Follower, election term: 3 09:09:24,387 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - leader314: Last sync leader does not match current leader follower315, need to catch up to -1 09:09:24,387 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader314: leader changed to follower315 09:09:24,387 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower315 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader314 updated: matchIndex: 0, nextIndex: 1 09:09:24,387 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorLeadershipTransferCohort - leader314: Successfully transferred leadership to follower315 in 18.32 ms 09:09:24,387 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader314: leader transfer succeeded - sending PoisonPill 09:09:24,387 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower leader314: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,387 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - leader314: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) [WARN] [07/01/2025 21:09:24.387] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader314] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader314, initialSyncDone=false] 09:09:24,388 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PekkoRaftStorage - leader314: terminating thread VirtualThread[#727,leader314-writer-143]/waiting 09:09:24,388 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: received 1 commands after 63.59 ms 09:09:24,388 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: completed 1 commands in 24.88 μs 09:09:24,388 PM [leader314-writer-143] [DEBUG] JournalWriteTask - leader314: journal writer stopped 09:09:24,388 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader314: journal closed 09:09:24,388 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRaftStorage - leader314: journal closed 09:09:24,388 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader314: stopped executor with 0 remaining tasks 09:09:24,388 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader314: stopped executor with 0 remaining tasks 09:09:24,388 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=3, votedFor=null 09:09:24,388 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower315 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower317 updated: matchIndex: 0, nextIndex: 1 09:09:24,388 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:24,388 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower317: Last sync leader does not match current leader follower315, need to catch up to -1 [WARN] [07/01/2025 21:09:24.388] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader314] received dead letter: FollowerInitialSyncUpStatus[memberId=leader314, initialSyncDone=true] 09:09:24,389 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) [WARN] [07/01/2025 21:09:24.389] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=false] [WARN] [07/01/2025 21:09:24.389] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=true] 09:09:24,905 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) 09:09:24,905 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower316: Last sync leader does not match current leader follower315, need to catch up to -1 [WARN] [07/01/2025 21:09:24.905] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader314] received dead letter from TestActor[pekko://test/user/follower315]: AppendEntries{term=3, leaderId=follower315, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:09:24,905 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower315 (Leader): sendAppendEntries: 0 is present for follower follower316 09:09:24,905 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower315 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=3, leaderId=follower315, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} [WARN] [07/01/2025 21:09:24.905] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=false] 09:09:24,905 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower316 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=follower315, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 09:09:24,905 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower316 (Follower): Number of entries to be appended = 1 09:09:24,905 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower316 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 09:09:24,905 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower316 (Follower): Append entry to log zero 09:09:24,905 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower316: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:09:24,906 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower316 (Follower): Log size is now 1 09:09:24,906 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower316 (Follower): Commit index set to 0 09:09:24,906 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower316 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower316, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:09:24,906 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower316: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) 09:09:24,906 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower316 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 09:09:24,906 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower316 (Follower): Setting last applied to 0 09:09:24,906 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower316: Moving last applied index from -1 to 0 09:09:24,906 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower316: Applying state for log index 0 data zero 09:09:24,906 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower316: applyState called: zero 09:09:24,906 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - follower315 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower316 updated: matchIndex: 0, nextIndex: 1 09:09:24,906 PM [follower316-writer-141] [DEBUG] JournalWriteTask - follower316: received 1 commands after 1.107 s 09:09:24,906 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:24.906] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=true] 09:09:24,911 PM [follower316-writer-141] [DEBUG] RaftStorageCompleter - follower316: 1 completion(s) pending 09:09:24,911 PM [follower316-writer-141] [DEBUG] JournalWriteTask - follower316: completed 1 commands in 5.250 ms 09:09:24,911 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower316: running 1 completion(s) 09:09:24,911 PM [follower316-writer-141] [DEBUG] JournalWriteTask - follower316: received 1 commands after 7.221 μs 09:09:24,911 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower316: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:09:24,911 PM [follower316-writer-141] [DEBUG] EntryJournalV1 - follower316: updated meta to JournalMeta[replayFrom=1, applyTo=1] 09:09:24,911 PM [follower316-writer-141] [DEBUG] RaftStorageCompleter - follower316: 1 completion(s) pending 09:09:24,911 PM [follower316-writer-141] [DEBUG] JournalWriteTask - follower316: completed 1 commands in 184.2 μs 09:09:24,911 PM [follower316-writer-141] [DEBUG] JournalWriteTask - follower316: waiting to receive commands 09:09:24,913 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower316: running 1 completion(s) 09:09:24,930 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower316: onLeaderTransitioning: LeaderTransitioning [leaderId=leader314] 09:09:24,930 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 09:09:24,930 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower316 starting 09:09:24,930 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower316: onShutDown 09:09:24,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower316: terminating thread VirtualThread[#725,follower316-writer-141]/waiting 09:09:24,931 PM [follower316-writer-141] [DEBUG] JournalWriteTask - follower316: received 1 commands after 19.39 ms 09:09:24,931 PM [follower316-writer-141] [DEBUG] JournalWriteTask - follower316: completed 1 commands in 32.95 μs 09:09:24,931 PM [follower316-writer-141] [DEBUG] JournalWriteTask - follower316: journal writer stopped 09:09:24,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower316: journal closed 09:09:24,931 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower316: journal closed 09:09:24,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower316: stopped executor with 0 remaining tasks 09:09:24,931 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower316: stopped executor with 0 remaining tasks 09:09:24,932 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower316 ending 09:09:24,932 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 09:09:24,932 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower315-notifier318#1046340623] 09:09:24,933 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower315-collector319#694943290] 09:09:24,933 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower315] 09:09:24,933 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower315: terminating thread VirtualThread[#724,follower315-writer-140]/waiting 09:09:24,933 PM [follower315-writer-140] [DEBUG] JournalWriteTask - follower315: received 1 commands after 608.5 ms 09:09:24,934 PM [follower315-writer-140] [DEBUG] JournalWriteTask - follower315: completed 1 commands in 21.94 μs 09:09:24,934 PM [follower315-writer-140] [DEBUG] JournalWriteTask - follower315: journal writer stopped 09:09:24,934 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower315: journal closed 09:09:24,934 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower315: journal closed 09:09:24,934 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower315: stopped executor with 0 remaining tasks 09:09:24,934 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower315: stopped executor with 0 remaining tasks 09:09:24,934 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower316-notifier320#1275719976] 09:09:24,934 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower316-collector321#977300357] 09:09:24,934 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower316] 09:09:24,934 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower317-notifier322#-1204688789] 09:09:24,935 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower317-collector323#730310436] 09:09:24,935 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower317] 09:09:24,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower317: terminating thread VirtualThread[#726,follower317-writer-142]/waiting 09:09:24,935 PM [follower317-writer-142] [DEBUG] JournalWriteTask - follower317: received 1 commands after 609.9 ms 09:09:24,935 PM [follower317-writer-142] [DEBUG] JournalWriteTask - follower317: completed 1 commands in 21.14 μs 09:09:24,935 PM [follower317-writer-142] [DEBUG] JournalWriteTask - follower317: journal writer stopped 09:09:24,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower317: journal closed 09:09:24,935 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower317: journal closed 09:09:24,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower317: stopped executor with 0 remaining tasks 09:09:24,935 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower317: stopped executor with 0 remaining tasks 09:09:24,935 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader314-notifier324#2112660776] 09:09:24,936 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader314-collector325#-858879885] 09:09:24,936 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader314] 09:09:24,939 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [07/01/2025 21:09:24.940] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader326]: org.opendaylight.controller.cluster.common.actor.Monitor@721a9040 09:09:24,941 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - leader326: Starting recovery with journal batch size 1000 09:09:24,941 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader326: started executor 09:09:24,941 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader326: started executor 09:09:24,942 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader326: metadata journal open: replayFrom=1 applyTo=0 09:09:24,943 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader326: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:24,943 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader326: journal open: applyTo=0 09:09:24,943 PM [leader326-writer-144] [DEBUG] JournalWriteTask - leader326: journal writer started 09:09:24,943 PM [leader326-writer-144] [DEBUG] JournalWriteTask - leader326: waiting to receive commands 09:09:24,944 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5272136182612895346/leader326/TermInfo.properties does not exist 09:09:24,945 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader326: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5272136182612895346/leader326/journal-v1-1.log 09:09:24,945 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader326: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest5272136182612895346/leader326/metadata-v1-1.log 09:09:24,945 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader326: no eligible files found 09:09:24,945 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - leader326: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:24,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader326: Set currentTerm=0, votedFor=null 09:09:24,947 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - leader326: Local TermInfo store seeded with TermInfo{term=0} 09:09:24,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader326: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6c58899 09:09:24,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader326: Moving last applied index from -1 to -1 09:09:24,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - leader326: empty journal: appending 0 entries 09:09:24,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader326: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:24,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - leader326: journal recovery completedin 462.5 μs with journalIndex=1 09:09:24,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:09:24,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - leader326 (Follower): Received TimeoutNow - switching to Candidate 09:09:24,949 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader326 (Candidate): Election: Candidate has following voting peers: [] 09:09:24,952 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader326: Set currentTerm=1, votedFor=leader326 09:09:24,952 PM [test-pekko.actor.default-dispatcher-13] [INFO] Candidate - leader326 (Candidate): Starting new election term 1 09:09:24,952 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - leader326 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 09:09:24,952 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader326 (Candidate): Received ElectionTimeout 09:09:24,952 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader326 (Leader): Election: Leader has following peers: [] 09:09:24,952 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - leader326 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 09:09:24,952 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader326 starting 09:09:24,952 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader326: onShutDown 09:09:24,953 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PekkoRaftStorage - leader326: terminating thread VirtualThread[#728,leader326-writer-144]/waiting 09:09:24,953 PM [leader326-writer-144] [DEBUG] JournalWriteTask - leader326: received 1 commands after 10.05 ms 09:09:24,953 PM [leader326-writer-144] [DEBUG] JournalWriteTask - leader326: completed 1 commands in 22.65 μs 09:09:24,953 PM [leader326-writer-144] [DEBUG] JournalWriteTask - leader326: journal writer stopped 09:09:24,953 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader326: journal closed 09:09:24,953 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRaftStorage - leader326: journal closed 09:09:24,953 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader326: stopped executor with 0 remaining tasks 09:09:24,953 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader326: stopped executor with 0 remaining tasks 09:09:24,953 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader326 ending 09:09:24,953 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 09:09:24,953 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader326-collector330#1717646025] 09:09:24,954 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader326] 09:09:24,956 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 09:09:24,956 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [07/01/2025 21:09:24.958] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower332]: org.opendaylight.controller.cluster.common.actor.Monitor@4bb9b6d1 09:09:24,958 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - follower332: Starting recovery with journal batch size 1000 09:09:24,958 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: started executor 09:09:24,958 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: started executor 09:09:24,959 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower332: metadata journal open: replayFrom=1 applyTo=0 09:09:24,959 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower332: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:24,959 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRaftStorage - follower332: journal open: applyTo=0 09:09:24,959 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: journal writer started 09:09:24,959 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: waiting to receive commands 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower332/TermInfo.properties does not exist 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower332/journal-v1-1.log 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower332/metadata-v1-1.log 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: no eligible files found 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PekkoRecovery - follower332: SnapshotOffer called. 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PekkoRecovery - follower332: initializing from snapshot taken at 2025-07-01T21:09:24.956Z 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=1, votedFor=null 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - follower332: Updated server config: isVoting: true, peers: [PeerInfo [id=leader331, address=pekko://test/user/leader331, votingState=VOTING], PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=VOTING], PeerInfo [id=follower334, address=pekko://test/user/follower334, votingState=NON_VOTING]] 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - Recovery snapshot applied for follower332 in 57.62 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower332: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:24.956Z 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower332: support() called 09:09:24,960 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower332/snapshot-0000000068644E84-38FB6700.tmp [WARN] [07/01/2025 21:09:24.961] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower333]: org.opendaylight.controller.cluster.common.actor.Monitor@6bdc28cd 09:09:24,961 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower333: Starting recovery with journal batch size 1000 09:09:24,961 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower333: started executor 09:09:24,962 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower333: started executor 09:09:24,962 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower333: metadata journal open: replayFrom=1 applyTo=0 09:09:24,962 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower332/snapshot-0000000068644E84-38FB6700.v1 09:09:24,962 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower333: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:24,962 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower333: journal open: applyTo=0 09:09:24,962 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower332/snapshot-0000000068644E84-38FB6700.v1 09:09:24,962 PM [follower333-writer-146] [DEBUG] JournalWriteTask - follower333: journal writer started 09:09:24,962 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower332/journal-v1-1.log 09:09:24,962 PM [follower333-writer-146] [DEBUG] JournalWriteTask - follower333: waiting to receive commands 09:09:24,962 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower332/metadata-v1-1.log 09:09:24,962 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower332: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:24.956Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower332/snapshot-0000000068644E84-38FB6700.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:24,962 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower332: local snapshot saved in 2.333 ms, deleting Pekko-persisted snapshots 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower333/TermInfo.properties does not exist 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower333/journal-v1-1.log 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower333/metadata-v1-1.log 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: no eligible files found 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower333: SnapshotOffer called. 09:09:24,963 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower332: Recovery completed in 3.154 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower333: initializing from snapshot taken at 2025-07-01T21:09:24.956Z 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=1, votedFor=null 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PeerInfos - follower333: Updated server config: isVoting: true, peers: [PeerInfo [id=leader331, address=pekko://test/user/leader331, votingState=VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING], PeerInfo [id=follower334, address=pekko://test/user/follower334, votingState=NON_VOTING]] 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - Recovery snapshot applied for follower333 in 84.29 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower333: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:24.956Z 09:09:24,963 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower333: support() called 09:09:24,964 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower333/snapshot-0000000068644E84-38FB6700.tmp 09:09:24,964 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - follower334: Starting recovery with journal batch size 1000 [WARN] [07/01/2025 21:09:24.964] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower334]: org.opendaylight.controller.cluster.common.actor.Monitor@4eef9048 09:09:24,964 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: started executor 09:09:24,964 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: started executor 09:09:24,964 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower334: metadata journal open: replayFrom=1 applyTo=0 09:09:24,965 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower334: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:24,965 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - follower334: journal open: applyTo=0 09:09:24,965 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: journal writer started 09:09:24,965 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: waiting to receive commands 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower334/TermInfo.properties does not exist 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower334/journal-v1-1.log 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower334/metadata-v1-1.log 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower334: no eligible files found 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower334: SnapshotOffer called. 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower334: initializing from snapshot taken at 2025-07-01T21:09:24.956Z 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower334: Set currentTerm=1, votedFor=null 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower334: Updated server config: isVoting: false, peers: [PeerInfo [id=leader331, address=pekko://test/user/leader331, votingState=VOTING], PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING]] 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for follower334 in 53.04 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower334: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:24.956Z 09:09:24,965 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower334: support() called 09:09:24,966 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower334: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower334/snapshot-0000000068644E84-38FB6700.tmp 09:09:24,966 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=1, votedFor=null 09:09:24,966 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower332: Local TermInfo store seeded with TermInfo{term=1} 09:09:24,966 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower332: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@22918644 09:09:24,966 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower332: Moving last applied index from -1 to -1 09:09:24,966 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - leader331: Starting recovery with journal batch size 1000 09:09:24,966 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: started executor 09:09:24,966 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: started executor 09:09:24,966 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower333/snapshot-0000000068644E84-38FB6700.v1 09:09:24,966 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower333/snapshot-0000000068644E84-38FB6700.v1 09:09:24,967 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower333/journal-v1-1.log 09:09:24,967 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower333/metadata-v1-1.log 09:09:24,967 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:24.956Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower333/snapshot-0000000068644E84-38FB6700.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:24,967 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower333: local snapshot saved in 3.298 ms, deleting Pekko-persisted snapshots 09:09:24,967 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower332: empty journal: appending 0 entries 09:09:24,967 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower332: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:24,967 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower333: Recovery completed in 4.208 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:24,967 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower332: journal recovery completedin 1.212 ms with journalIndex=1 09:09:24,967 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower334: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower334/snapshot-0000000068644E84-38FB6700.v1 09:09:24,968 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower334: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower334/snapshot-0000000068644E84-38FB6700.v1 09:09:24,968 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower334/journal-v1-1.log 09:09:24,968 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower334/metadata-v1-1.log 09:09:24,968 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower334: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:24.956Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/follower334/snapshot-0000000068644E84-38FB6700.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:24,968 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower334: local snapshot saved in 2.245 ms, deleting Pekko-persisted snapshots 09:09:24,968 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower334: Recovery completed in 2.868 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:24,969 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower332: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) [WARN] [07/01/2025 21:09:24.966] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader331]: org.opendaylight.controller.cluster.common.actor.Monitor@6162678e 09:09:24,971 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader331: metadata journal open: replayFrom=1 applyTo=0 09:09:24,971 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader331: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:24,971 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - leader331: journal open: applyTo=0 09:09:24,971 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: journal writer started 09:09:24,971 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: waiting to receive commands 09:09:24,971 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/leader331/TermInfo.properties does not exist 09:09:24,971 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/leader331/journal-v1-1.log 09:09:24,971 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/leader331/metadata-v1-1.log 09:09:24,972 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: no eligible files found 09:09:24,972 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - leader331: SnapshotOffer called. 09:09:24,972 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - leader331: initializing from snapshot taken at 2025-07-01T21:09:24.956Z 09:09:24,972 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader331: Set currentTerm=1, votedFor=null 09:09:24,972 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - leader331: Updated server config: isVoting: true, peers: [PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING], PeerInfo [id=follower334, address=pekko://test/user/follower334, votingState=NON_VOTING]] 09:09:24,972 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - Recovery snapshot applied for leader331 in 52.57 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:24,972 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - leader331: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:24.956Z 09:09:24,972 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader331: support() called 09:09:24,972 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/leader331/snapshot-0000000068644E84-38FB6700.tmp 09:09:24,973 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=1, votedFor=null 09:09:24,973 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower333: Local TermInfo store seeded with TermInfo{term=1} 09:09:24,973 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower333: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6ca8c8e 09:09:24,974 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower333: Moving last applied index from -1 to -1 09:09:24,974 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower333: empty journal: appending 0 entries 09:09:24,974 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower333: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:24,974 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower333: journal recovery completedin 303.8 μs with journalIndex=1 09:09:24,974 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower333: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:24,976 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower334: Set currentTerm=1, votedFor=null 09:09:24,976 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower334: Local TermInfo store seeded with TermInfo{term=1} 09:09:24,976 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower334: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@3de8cbe6 09:09:24,976 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower334: Moving last applied index from -1 to -1 09:09:24,976 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower334: empty journal: appending 0 entries 09:09:24,976 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower334: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:24,976 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower334: journal recovery completedin 209.7 μs with journalIndex=1 09:09:24,976 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower334: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:24,976 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/leader331/snapshot-0000000068644E84-38FB6700.v1 09:09:24,976 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/leader331/snapshot-0000000068644E84-38FB6700.v1 09:09:24,976 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/leader331/journal-v1-1.log 09:09:24,976 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/leader331/metadata-v1-1.log 09:09:24,976 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader331: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:24.956Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest18427193616847524467/leader331/snapshot-0000000068644E84-38FB6700.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:24,977 PM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - leader331: local snapshot saved in 4.820 ms, deleting Pekko-persisted snapshots 09:09:24,977 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - leader331: Recovery completed in 5.819 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:24,979 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - leader331: Set currentTerm=1, votedFor=null 09:09:24,979 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - leader331: Local TermInfo store seeded with TermInfo{term=1} 09:09:24,979 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader331: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@1c378e86 09:09:24,979 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader331: Moving last applied index from -1 to -1 09:09:24,980 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - leader331: empty journal: appending 0 entries 09:09:24,980 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - leader331: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:24,980 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - leader331: journal recovery completedin 226.8 μs with journalIndex=1 09:09:24,980 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader331: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:25,304 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:09:25,304 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - leader331 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.150:17355, Removed) 09:09:25,304 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - leader331 (Follower): Received ElectionTimeout - switching to Candidate 09:09:25,304 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - leader331 (Candidate): Election: Candidate has following voting peers: [follower333, follower332] 09:09:25,309 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - leader331: Set currentTerm=2, votedFor=leader331 09:09:25,309 PM [test-pekko.actor.default-dispatcher-7] [INFO] Candidate - leader331 (Candidate): Starting new election term 2 09:09:25,309 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - leader331 (Candidate): Sending RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} to peer follower333 09:09:25,309 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Candidate - leader331 (Candidate): Sending RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} to peer follower332 09:09:25,309 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower332 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:25,309 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower333 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:25,309 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower332 (Follower): Cluster state: [] 09:09:25,309 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower332 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:25,309 PM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower332 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:25,309 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower333 (Follower): Cluster state: [] 09:09:25,309 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower333 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:25,309 PM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower333 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:25,310 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActorBehavior - leader331 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:09:25,312 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=2, votedFor=null 09:09:25,312 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower332 (Follower): In requestVote: RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:25,313 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=2, votedFor=null 09:09:25,313 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower333 (Follower): In requestVote: RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:25,314 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=2, votedFor=leader331 09:09:25,314 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower332 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:09:25,314 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader331 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:09:25,314 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader331 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:09:25,314 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Election: Leader has following peers: [follower333, follower332, follower334] 09:09:25,315 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:25,315 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:25,315 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:25,315 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader331 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:09:25,315 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower332: Last sync leader does not match current leader leader331, need to catch up to -1 09:09:25,315 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower332 updated: matchIndex: -1, nextIndex: 0 09:09:25,315 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:25,315 PM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower334 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term [WARN] [07/01/2025 21:09:25.315] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower332] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower332, initialSyncDone=false] 09:09:25,317 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=2, votedFor=leader331 09:09:25,317 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower333 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:09:25,317 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower333: Last sync leader does not match current leader leader331, need to catch up to -1 [WARN] [07/01/2025 21:09:25.317] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower333] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower333, initialSyncDone=false] 09:09:25,318 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower333 updated: matchIndex: -1, nextIndex: 0 09:09:25,318 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:25,318 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower334: Set currentTerm=2, votedFor=null 09:09:25,318 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower334: Last sync leader does not match current leader leader331, need to catch up to -1 09:09:25,318 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower334 updated: matchIndex: -1, nextIndex: 0 09:09:25,318 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:25.318] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower334] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower334, initialSyncDone=false] 09:09:25,335 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:09:25,335 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 09:09:25,336 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 09:09:25,336 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader331: Persist data index=0 term=2 command=zero 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader331: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 09:09:25,336 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: received 1 commands after 365.1 ms 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower333: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower332 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower332: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:25,336 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower334 09:09:25,337 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower334: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:09:25,337 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader331: awaiting more completions to resolve 1 synchronized callback(s) 09:09:25,337 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower332 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:09:25,337 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower332 (Follower): Number of entries to be appended = 1 09:09:25,337 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower332 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 09:09:25,337 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower332 (Follower): Append entry to log zero 09:09:25,337 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:09:25,337 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower332: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:09:25,337 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): Number of entries to be appended = 1 09:09:25,337 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 09:09:25,337 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): Append entry to log zero 09:09:25,337 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower334: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:09:25,337 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower332 (Follower): Log size is now 1 09:09:25,337 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower332 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower332, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:09:25,337 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached -1 (of expected -1) 09:09:25,337 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: received 1 commands after 377.7 ms [WARN] [07/01/2025 21:09:25.337] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower332] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower332, initialSyncDone=true] 09:09:25,337 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): Log size is now 1 09:09:25,337 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower334, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:09:25,337 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached -1 (of expected -1) 09:09:25,337 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: received 1 commands after 372.6 ms [WARN] [07/01/2025 21:09:25.338] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower334] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower334, initialSyncDone=true] 09:09:25,342 PM [leader331-writer-148] [DEBUG] RaftStorageCompleter - leader331: 1 completion(s) pending 09:09:25,342 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: completed 1 commands in 5.505 ms 09:09:25,342 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader331: running 1 completion(s) 09:09:25,342 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: waiting to receive commands 09:09:25,342 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader331: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:09:25,342 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader331: completed synchronized callback AppendSubmittedCallback{entry=JournaledLogEntry{index=0, term=2, command=zero, pending=false}, callback=org.opendaylight.controller.cluster.raft.RaftActor$$Lambda/0x00000007c0879d90@cd4c4a8} 09:09:25,342 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower332 updated: matchIndex: 0, nextIndex: 1 09:09:25,342 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 09:09:25,342 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader331 (Leader): Setting last applied to 0 09:09:25,342 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader331: Moving last applied index from -1 to 0 09:09:25,342 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader331: Applying state for log index 0 data zero 09:09:25,342 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader331: applyState called: zero 09:09:25,342 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:25,342 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: received 1 commands after 632.9 μs 09:09:25,343 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower334 updated: matchIndex: 0, nextIndex: 1 09:09:25,343 PM [leader331-writer-148] [DEBUG] EntryJournalV1 - leader331: updated meta to JournalMeta[replayFrom=1, applyTo=1] 09:09:25,343 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:25,343 PM [leader331-writer-148] [DEBUG] RaftStorageCompleter - leader331: 1 completion(s) pending 09:09:25,343 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: completed 1 commands in 229.2 μs 09:09:25,343 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: waiting to receive commands 09:09:25,343 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader331: running 1 completion(s) 09:09:25,343 PM [follower334-writer-147] [DEBUG] RaftStorageCompleter - follower334: 1 completion(s) pending 09:09:25,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower334: running 1 completion(s) 09:09:25,343 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: completed 1 commands in 5.480 ms 09:09:25,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower334: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:09:25,343 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: waiting to receive commands 09:09:25,343 PM [follower332-writer-145] [DEBUG] RaftStorageCompleter - follower332: 1 completion(s) pending 09:09:25,343 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower332 (Follower): Commit index set to 0 09:09:25,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower334 (Follower): Commit index set to 0 09:09:25,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:09:25,343 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:09:25,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower334 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 09:09:25,343 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: completed 1 commands in 6.178 ms 09:09:25,343 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Follower - follower332 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 09:09:25,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower334 (Follower): Setting last applied to 0 09:09:25,343 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: waiting to receive commands 09:09:25,343 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower332 (Follower): Setting last applied to 0 09:09:25,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower334: Moving last applied index from -1 to 0 09:09:25,343 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower332: Moving last applied index from -1 to 0 09:09:25,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower334: Applying state for log index 0 data zero 09:09:25,343 PM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower334: applyState called: zero 09:09:25,343 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower332: Applying state for log index 0 data zero 09:09:25,343 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower332: applyState called: zero 09:09:25,344 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: received 1 commands after 555.5 μs 09:09:25,344 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: received 1 commands after 386.7 μs 09:09:25,344 PM [follower332-writer-145] [DEBUG] EntryJournalV1 - follower332: updated meta to JournalMeta[replayFrom=1, applyTo=1] 09:09:25,344 PM [follower332-writer-145] [DEBUG] RaftStorageCompleter - follower332: 1 more completion(s) pending 09:09:25,344 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: completed 1 commands in 191.3 μs 09:09:25,344 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: waiting to receive commands 09:09:25,344 PM [follower334-writer-147] [DEBUG] EntryJournalV1 - follower334: updated meta to JournalMeta[replayFrom=1, applyTo=1] 09:09:25,344 PM [follower334-writer-147] [DEBUG] RaftStorageCompleter - follower334: 1 completion(s) pending 09:09:25,344 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: completed 1 commands in 293.6 μs 09:09:25,344 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: waiting to receive commands 09:09:25,344 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorageCompleter - follower332: running 2 completion(s) 09:09:25,344 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower334: running 1 completion(s) 09:09:25,344 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] ReplicatedLogImpl - follower332: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:09:25,387 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 09:09:25,387 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 09:09:25,387 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 09:09:25,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader331: onRequestLeadership RequestLeadership [requestedFollowerId=follower333, replyTo=Actor[pekko://test/user/$c#446728115]] 09:09:25,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader331: Initiating leader transfer 09:09:25,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorLeadershipTransferCohort - leader331: pauseLeader successfully completed - doing transfer 09:09:25,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Leader - leader331 (Leader): Attempting to transfer leadership 09:09:25,387 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower332: onLeaderTransitioning: LeaderTransitioning [leaderId=leader331] 09:09:25,387 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower334: onLeaderTransitioning: LeaderTransitioning [leaderId=leader331] 09:09:25,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:25,387 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 09:09:25,388 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower333: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 09:09:25,388 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower333: onLeaderTransitioning: LeaderTransitioning [leaderId=leader331] 09:09:25,388 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:25,388 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:09:25,388 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:09:25,388 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:09:25,434 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 09:09:25,554 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 09:09:25,554 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower333: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 09:09:25,555 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:09:25,555 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:09:25,674 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:09:25,674 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:09:25,794 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorLeadershipTransferCohort - leader331: leader transfer aborted 09:09:25,794 PM [test-pekko.actor.default-dispatcher-7] [WARN] RaftActorLeadershipTransferCohort - leader331: Failed to transfer leadership in 406.6 ms 09:09:25,794 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader331: LeadershipTransfer request from follower333 failed 09:09:25,794 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] Leader - leader331 (Leader): Leadership transfer expired 09:09:25,794 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:09:25,794 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:09:25,842 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 09:09:25,842 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower332-notifier335#105890566] 09:09:25,854 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower332-collector336#94427653] 09:09:25,854 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower332] 09:09:25,854 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRaftStorage - follower332: terminating thread VirtualThread[#729,follower332-writer-145]/waiting 09:09:25,854 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: received 1 commands after 510.5 ms 09:09:25,854 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: completed 1 commands in 39.13 μs 09:09:25,854 PM [follower332-writer-145] [DEBUG] JournalWriteTask - follower332: journal writer stopped 09:09:25,855 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower332: journal closed 09:09:25,855 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - follower332: journal closed 09:09:25,855 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower332: stopped executor with 0 remaining tasks 09:09:25,855 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower332: stopped executor with 0 remaining tasks 09:09:25,855 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower333-notifier337#-1858787056] 09:09:25,855 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower333-collector338#664102667] 09:09:25,856 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower333] 09:09:25,856 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRaftStorage - follower333: terminating thread VirtualThread[#730,follower333-writer-146]/waiting 09:09:25,856 PM [follower333-writer-146] [DEBUG] JournalWriteTask - follower333: received 1 commands after 893.5 ms 09:09:25,856 PM [follower333-writer-146] [DEBUG] JournalWriteTask - follower333: completed 1 commands in 21.75 μs 09:09:25,856 PM [follower333-writer-146] [DEBUG] JournalWriteTask - follower333: journal writer stopped 09:09:25,856 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower333: journal closed 09:09:25,856 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - follower333: journal closed 09:09:25,856 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: stopped executor with 0 remaining tasks 09:09:25,856 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower333: stopped executor with 0 remaining tasks 09:09:25,856 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower334-notifier339#-1986450016] 09:09:25,857 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower334-collector340#1391973529] 09:09:25,857 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower334] 09:09:25,857 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRaftStorage - follower334: terminating thread VirtualThread[#731,follower334-writer-147]/waiting 09:09:25,857 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: received 1 commands after 513.1 ms 09:09:25,857 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: completed 1 commands in 22.06 μs 09:09:25,857 PM [follower334-writer-147] [DEBUG] JournalWriteTask - follower334: journal writer stopped 09:09:25,857 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower334: journal closed 09:09:25,857 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - follower334: journal closed 09:09:25,857 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower334: stopped executor with 0 remaining tasks 09:09:25,857 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower334: stopped executor with 0 remaining tasks 09:09:25,857 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader331-notifier341#1082585101] 09:09:25,858 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader331-collector342#1763697529] 09:09:25,858 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader331] 09:09:25,858 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRaftStorage - leader331: terminating thread VirtualThread[#732,leader331-writer-148]/waiting 09:09:25,858 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: received 1 commands after 515.3 ms 09:09:25,858 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: completed 1 commands in 21.92 μs 09:09:25,858 PM [leader331-writer-148] [DEBUG] JournalWriteTask - leader331: journal writer stopped 09:09:25,858 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - leader331: journal closed 09:09:25,858 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - leader331: journal closed 09:09:25,858 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader331: stopped executor with 0 remaining tasks 09:09:25,858 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader331: stopped executor with 0 remaining tasks 09:09:25,859 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$c#446728115] 09:09:25,861 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 09:09:25,861 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [07/01/2025 21:09:25.865] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower344]: org.opendaylight.controller.cluster.common.actor.Monitor@1fbbf56a 09:09:25,865 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActor - follower344: Starting recovery with journal batch size 1000 09:09:25,865 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: started executor 09:09:25,866 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: started executor 09:09:25,866 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower344: metadata journal open: replayFrom=1 applyTo=0 09:09:25,866 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower344: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:25,866 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - follower344: journal open: applyTo=0 09:09:25,867 PM [follower344-writer-149] [DEBUG] JournalWriteTask - follower344: journal writer started 09:09:25,867 PM [follower344-writer-149] [DEBUG] JournalWriteTask - follower344: waiting to receive commands 09:09:25,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower344/TermInfo.properties does not exist 09:09:25,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower344/journal-v1-1.log 09:09:25,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower344/metadata-v1-1.log 09:09:25,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: no eligible files found 09:09:25,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower344: SnapshotOffer called. 09:09:25,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower344: initializing from snapshot taken at 2025-07-01T21:09:25.861Z 09:09:25,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=1, votedFor=null 09:09:25,867 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PeerInfos - follower344: Updated server config: isVoting: true, peers: [PeerInfo [id=leader343, address=pekko://test/user/leader343, votingState=VOTING], PeerInfo [id=follower346, address=pekko://test/user/follower346, votingState=NON_VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=VOTING]] [WARN] [07/01/2025 21:09:25.868] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower345]: org.opendaylight.controller.cluster.common.actor.Monitor@22e4dcbd 09:09:25,868 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - Recovery snapshot applied for follower344 in 761.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:25,868 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower344: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:25.861Z 09:09:25,868 PM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower344: support() called 09:09:25,869 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower344/snapshot-0000000068644E85-3351D140.tmp 09:09:25,869 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - follower345: Starting recovery with journal batch size 1000 09:09:25,869 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower345: started executor 09:09:25,869 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower345: started executor 09:09:25,869 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower345: metadata journal open: replayFrom=1 applyTo=0 09:09:25,870 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower345: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:25,870 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRaftStorage - follower345: journal open: applyTo=0 09:09:25,870 PM [follower345-writer-150] [DEBUG] JournalWriteTask - follower345: journal writer started 09:09:25,870 PM [follower345-writer-150] [DEBUG] JournalWriteTask - follower345: waiting to receive commands 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower345/TermInfo.properties does not exist 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower345/journal-v1-1.log 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower345/metadata-v1-1.log 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: no eligible files found 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower345: SnapshotOffer called. 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - follower345: initializing from snapshot taken at 2025-07-01T21:09:25.861Z 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=1, votedFor=null 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - follower345: Updated server config: isVoting: true, peers: [PeerInfo [id=leader343, address=pekko://test/user/leader343, votingState=VOTING], PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower346, address=pekko://test/user/follower346, votingState=NON_VOTING]] 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for follower345 in 267.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower345: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:25.861Z 09:09:25,871 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - follower345: support() called 09:09:25,872 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower345/snapshot-0000000068644E85-3351D140.tmp 09:09:25,872 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower344/snapshot-0000000068644E85-3351D140.v1 09:09:25,872 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower344/journal-v1-1.log 09:09:25,872 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower344/metadata-v1-1.log 09:09:25,872 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower344/snapshot-0000000068644E85-3351D140.v1 09:09:25,872 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower344: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:25.861Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower344/snapshot-0000000068644E85-3351D140.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:25,873 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower344: local snapshot saved in 4.117 ms, deleting Pekko-persisted snapshots [WARN] [07/01/2025 21:09:25.873] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower346]: org.opendaylight.controller.cluster.common.actor.Monitor@1c5e0424 09:09:25,873 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower344: Recovery completed in 5.855 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:25,874 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower345/snapshot-0000000068644E85-3351D140.v1 09:09:25,874 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower345/journal-v1-1.log 09:09:25,874 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower345/metadata-v1-1.log 09:09:25,874 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower345/snapshot-0000000068644E85-3351D140.v1 09:09:25,874 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:25.861Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower345/snapshot-0000000068644E85-3351D140.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:25,874 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower345: local snapshot saved in 2.920 ms, deleting Pekko-persisted snapshots 09:09:25,874 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - follower346: Starting recovery with journal batch size 1000 09:09:25,875 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: started executor 09:09:25,875 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower345: Recovery completed in 3.847 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:25,875 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: started executor 09:09:25,875 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower346: metadata journal open: replayFrom=1 applyTo=0 09:09:25,876 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower346: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:25,876 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRaftStorage - follower346: journal open: applyTo=0 09:09:25,876 PM [follower346-writer-151] [DEBUG] JournalWriteTask - follower346: journal writer started 09:09:25,876 PM [follower346-writer-151] [DEBUG] JournalWriteTask - follower346: waiting to receive commands 09:09:25,876 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower346/TermInfo.properties does not exist 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower346/journal-v1-1.log 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower346/metadata-v1-1.log 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: no eligible files found 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PekkoRecovery - follower346: SnapshotOffer called. 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PekkoRecovery - follower346: initializing from snapshot taken at 2025-07-01T21:09:25.861Z 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower346: Set currentTerm=1, votedFor=null 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - follower346: Updated server config: isVoting: false, peers: [PeerInfo [id=leader343, address=pekko://test/user/leader343, votingState=VOTING], PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=VOTING]] 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - Recovery snapshot applied for follower346 in 71.33 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower346: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:25.861Z 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower346: support() called 09:09:25,877 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower346/snapshot-0000000068644E85-3351D140.tmp [WARN] [07/01/2025 21:09:25.877] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader343]: org.opendaylight.controller.cluster.common.actor.Monitor@16a5691d 09:09:25,878 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - leader343: Starting recovery with journal batch size 1000 09:09:25,878 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: started executor 09:09:25,878 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: started executor 09:09:25,878 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader343: metadata journal open: replayFrom=1 applyTo=0 09:09:25,879 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=1, votedFor=null 09:09:25,879 PM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower344: Local TermInfo store seeded with TermInfo{term=1} 09:09:25,879 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower344: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@1ecc8e65 09:09:25,879 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower344: Moving last applied index from -1 to -1 09:09:25,879 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower344: empty journal: appending 0 entries 09:09:25,879 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower344: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:25,879 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower344: journal recovery completedin 240.2 μs with journalIndex=1 09:09:25,879 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader343: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:09:25,879 PM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - leader343: journal open: applyTo=0 09:09:25,879 PM [leader343-writer-152] [DEBUG] JournalWriteTask - leader343: journal writer started 09:09:25,879 PM [leader343-writer-152] [DEBUG] JournalWriteTask - leader343: waiting to receive commands 09:09:25,879 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower344: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:25,880 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower346/snapshot-0000000068644E85-3351D140.v1 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/leader343/TermInfo.properties does not exist 09:09:25,880 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower346/journal-v1-1.log 09:09:25,880 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower346/metadata-v1-1.log 09:09:25,880 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower346/snapshot-0000000068644E85-3351D140.v1 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/leader343/journal-v1-1.log 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/leader343/metadata-v1-1.log 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader343: no eligible files found 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - leader343: SnapshotOffer called. 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - leader343: initializing from snapshot taken at 2025-07-01T21:09:25.861Z 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=1, votedFor=null 09:09:25,880 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower346: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:25.861Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/follower346/snapshot-0000000068644E85-3351D140.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:25,880 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower346: local snapshot saved in 2.978 ms, deleting Pekko-persisted snapshots 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - leader343: Updated server config: isVoting: true, peers: [PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower346, address=pekko://test/user/follower346, votingState=NON_VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=VOTING]] 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for leader343 in 81.56 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader343: migrating from Pekko persistent-snapshot taken at 2025-07-01T21:09:25.861Z 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader343: support() called 09:09:25,880 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader343: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/leader343/snapshot-0000000068644E85-3351D140.tmp 09:09:25,880 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower346: Recovery completed in 3.692 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:25,881 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=1, votedFor=null 09:09:25,881 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower345: Local TermInfo store seeded with TermInfo{term=1} 09:09:25,881 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower345: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@23264939 09:09:25,881 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower345: Moving last applied index from -1 to -1 09:09:25,881 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower345: empty journal: appending 0 entries 09:09:25,882 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower345: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:25,882 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower345: journal recovery completedin 255.3 μs with journalIndex=1 09:09:25,882 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower345: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:25,882 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader343: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/leader343/snapshot-0000000068644E85-3351D140.v1 09:09:25,882 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/leader343/journal-v1-1.log 09:09:25,882 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/leader343/metadata-v1-1.log 09:09:25,882 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader343: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/leader343/snapshot-0000000068644E85-3351D140.v1 09:09:25,882 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader343: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-07-01T21:09:25.861Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6504379894636920398/leader343/snapshot-0000000068644E85-3351D140.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:09:25,882 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader343: local snapshot saved in 2.267 ms, deleting Pekko-persisted snapshots 09:09:25,883 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader343: Recovery completed in 2.847 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:09:25,883 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower346: Set currentTerm=1, votedFor=null 09:09:25,883 PM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower346: Local TermInfo store seeded with TermInfo{term=1} 09:09:25,883 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower346: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@17d1c28f 09:09:25,883 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower346: Moving last applied index from -1 to -1 09:09:25,883 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower346: empty journal: appending 0 entries 09:09:25,883 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower346: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:25,883 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower346: journal recovery completedin 268.5 μs with journalIndex=1 09:09:25,884 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower346: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:25,884 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=1, votedFor=null 09:09:25,884 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader343: Local TermInfo store seeded with TermInfo{term=1} 09:09:25,884 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader343: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@1d3d549a 09:09:25,884 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader343: Moving last applied index from -1 to -1 09:09:25,885 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader343: empty journal: appending 0 entries 09:09:25,885 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader343: updated meta to JournalMeta[replayFrom=1, applyTo=0] 09:09:25,885 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - leader343: journal recovery completedin 236.2 μs with journalIndex=1 09:09:25,885 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader343: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:09:26,213 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:09:26,213 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - leader343 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.150:17355, Removed) 09:09:26,214 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - leader343 (Follower): Received ElectionTimeout - switching to Candidate 09:09:26,214 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader343 (Candidate): Election: Candidate has following voting peers: [follower344, follower345] 09:09:26,217 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=2, votedFor=leader343 09:09:26,217 PM [test-pekko.actor.default-dispatcher-13] [INFO] Candidate - leader343 (Candidate): Starting new election term 2 09:09:26,217 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader343 (Candidate): Sending RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} to peer follower344 09:09:26,217 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader343 (Candidate): Sending RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} to peer follower345 09:09:26,217 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - leader343 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:09:26,217 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower345 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:26,217 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower345 (Follower): Cluster state: [] 09:09:26,217 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower345 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:26,217 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower345 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:26,218 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower344 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:09:26,218 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower344 (Follower): Cluster state: [] 09:09:26,218 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower344 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:26,218 PM [test-pekko.actor.default-dispatcher-7] [INFO] Follower - follower344 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:09:26,220 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=2, votedFor=null 09:09:26,220 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower345 (Follower): In requestVote: RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:26,222 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=2, votedFor=null 09:09:26,222 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower344 (Follower): In requestVote: RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:26,223 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=2, votedFor=leader343 09:09:26,223 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower345 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:09:26,223 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader343 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:09:26,223 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader343 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:09:26,223 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Election: Leader has following peers: [follower344, follower346, follower345] 09:09:26,224 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:26,224 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:26,224 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=2, votedFor=leader343 09:09:26,224 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:26,224 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower344 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:09:26,224 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader343 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:09:26,224 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower346 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 09:09:26,224 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower345: Last sync leader does not match current leader leader343, need to catch up to -1 09:09:26,224 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower345 updated: matchIndex: -1, nextIndex: 0 [WARN] [07/01/2025 21:09:26.224] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower345] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower345, initialSyncDone=false] 09:09:26,224 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:26,224 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower344: Last sync leader does not match current leader leader343, need to catch up to -1 09:09:26,224 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader343 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower344 updated: matchIndex: -1, nextIndex: 0 09:09:26,224 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:26.225] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower344] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower344, initialSyncDone=false] 09:09:26,226 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower346: Set currentTerm=2, votedFor=null 09:09:26,226 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower346: Last sync leader does not match current leader leader343, need to catch up to -1 09:09:26,227 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader343 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower346 updated: matchIndex: -1, nextIndex: 0 [WARN] [07/01/2025 21:09:26.227] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower346] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower346, initialSyncDone=false] 09:09:26,227 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:26,239 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:09:26,239 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 09:09:26,240 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 09:09:26,240 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower344] 09:09:26,250 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower344: terminating thread VirtualThread[#733,follower344-writer-149]/waiting 09:09:26,251 PM [follower344-writer-149] [DEBUG] JournalWriteTask - follower344: received 1 commands after 383.9 ms 09:09:26,251 PM [follower344-writer-149] [DEBUG] JournalWriteTask - follower344: completed 1 commands in 38.72 μs 09:09:26,251 PM [follower344-writer-149] [DEBUG] JournalWriteTask - follower344: journal writer stopped 09:09:26,251 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower344: journal closed 09:09:26,251 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower344: journal closed 09:09:26,251 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: stopped executor with 0 remaining tasks 09:09:26,251 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: stopped executor with 0 remaining tasks 09:09:26,251 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower346] 09:09:26,262 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower346: terminating thread VirtualThread[#735,follower346-writer-151]/waiting 09:09:26,262 PM [follower346-writer-151] [DEBUG] JournalWriteTask - follower346: received 1 commands after 386.5 ms 09:09:26,262 PM [follower346-writer-151] [DEBUG] JournalWriteTask - follower346: completed 1 commands in 21.11 μs 09:09:26,262 PM [follower346-writer-151] [DEBUG] JournalWriteTask - follower346: journal writer stopped 09:09:26,262 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower346: journal closed 09:09:26,262 PM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower346: journal closed 09:09:26,262 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower346: stopped executor with 0 remaining tasks 09:09:26,262 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower346: stopped executor with 0 remaining tasks 09:09:26,263 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 09:09:26,263 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 09:09:26,263 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader343: onRequestLeadership RequestLeadership [requestedFollowerId=follower345, replyTo=Actor[pekko://test/user/$d#-265560408]] 09:09:26,263 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader343: Initiating leader transfer 09:09:26,263 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader343: pauseLeader successfully completed - doing transfer 09:09:26,263 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] Leader - leader343 (Leader): Attempting to transfer leadership [WARN] [07/01/2025 21:09:26.263] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/leader343]: LeaderTransitioning [leaderId=leader343] [WARN] [07/01/2025 21:09:26.263] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/leader343]: LeaderTransitioning [leaderId=leader343] 09:09:26,263 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:26,263 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower345: onLeaderTransitioning: LeaderTransitioning [leaderId=leader343] 09:09:26,263 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:26.263] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:09:26,264 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:26.264] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:09:26,264 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower345: Lagging 0 entries behind leader leader343 and reached -1 (of expected -1) 09:09:26,264 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader343 (Leader): tryToCompleteLeadershipTransfer: followerId: follower345, matchIndex: -1, lastIndex: -1, isVoting: true 09:09:26,264 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader343 (Leader): Follower's log matches - sending ElectionTimeout 09:09:26,264 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:26.264] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower345] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower345, initialSyncDone=true] 09:09:26,264 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:26.264] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:09:26,264 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:26.264] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:09:26,264 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader343 (Leader): Leader transfer complete 09:09:26,264 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader343: leader transfer complete - waiting for new leader 09:09:26,264 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower345: Lagging 0 entries behind leader leader343 and reached -1 (of expected -1) 09:09:26,264 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:09:26,264 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower345 (Follower): Received TimeoutNow - switching to Candidate 09:09:26,265 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - follower345 (Candidate): Election: Candidate has following voting peers: [leader343, follower344] 09:09:26,267 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=3, votedFor=follower345 09:09:26,267 PM [test-pekko.actor.default-dispatcher-13] [INFO] Candidate - follower345 (Candidate): Starting new election term 3 09:09:26,267 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - follower345 (Candidate): Sending RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} to peer leader343 09:09:26,267 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - follower345 (Candidate): Sending RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} to peer follower344 09:09:26,267 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - follower345 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 09:09:26,268 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader343 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. [WARN] [07/01/2025 21:09:26.268] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/follower345]: RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} 09:09:26,268 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader343 (Leader): Cluster state: [] 09:09:26,268 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader343 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:09:26,268 PM [test-pekko.actor.default-dispatcher-7] [INFO] AbstractLeader - leader343 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower 09:09:26,270 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=3, votedFor=null 09:09:26,270 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractLeader - leader343 (Leader): Leadership transfer in progress - processing RequestVote 09:09:26,270 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader343 (Leader): In requestVote: RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 09:09:26,272 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=3, votedFor=follower345 09:09:26,272 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - leader343 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 09:09:26,272 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - follower345 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 09:09:26,272 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - follower345 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 09:09:26,272 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActorBehavior - leader343 (Leader) :- Switching from behavior Leader to Follower, election term: 3 09:09:26,272 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower345 (Leader): Election: Leader has following peers: [leader343, follower344, follower346] 09:09:26,272 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower leader343: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:26,272 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower follower344: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:26,273 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - leader343: Last sync leader does not match current leader follower345, need to catch up to -1 09:09:26,273 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower follower346: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:09:26,273 PM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorBehavior - follower345 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [07/01/2025 21:09:26.273] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/follower345]: AppendEntries{term=3, leaderId=follower345, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} [WARN] [07/01/2025 21:09:26.273] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/follower345]: AppendEntries{term=3, leaderId=follower345, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 09:09:26,273 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorLeadershipTransferCohort - leader343: leader changed to follower345 09:09:26,273 PM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActorLeadershipTransferCohort - leader343: Successfully transferred leadership to follower345 in 9.848 ms 09:09:26,273 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader343: Leadership transferred successfully to follower345 09:09:26,273 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower345 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader343 updated: matchIndex: -1, nextIndex: 0 09:09:26,273 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower leader343: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [07/01/2025 21:09:26.273] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/leader343] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader343, initialSyncDone=false] 09:09:26,314 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 09:09:26,314 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower344-notifier347#-1002337180] 09:09:26,325 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower344-collector348#-554904202] 09:09:26,326 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower345-notifier349#-291939431] 09:09:26,326 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower345-collector350#-1789995591] 09:09:26,326 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower345] 09:09:26,326 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRaftStorage - follower345: terminating thread VirtualThread[#734,follower345-writer-150]/waiting 09:09:26,326 PM [follower345-writer-150] [DEBUG] JournalWriteTask - follower345: received 1 commands after 456.6 ms 09:09:26,326 PM [follower345-writer-150] [DEBUG] JournalWriteTask - follower345: completed 1 commands in 24.54 μs 09:09:26,326 PM [follower345-writer-150] [DEBUG] JournalWriteTask - follower345: journal writer stopped 09:09:26,327 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower345: journal closed 09:09:26,327 PM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - follower345: journal closed 09:09:26,327 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower345: stopped executor with 0 remaining tasks 09:09:26,327 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower345: stopped executor with 0 remaining tasks 09:09:26,327 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower346-notifier351#1514179856] 09:09:26,327 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower346-collector352#688004119] 09:09:26,327 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader343-notifier353#-15113877] 09:09:26,327 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader343-collector354#1423540695] 09:09:26,328 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader343] 09:09:26,328 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRaftStorage - leader343: terminating thread VirtualThread[#736,leader343-writer-152]/waiting 09:09:26,328 PM [leader343-writer-152] [DEBUG] JournalWriteTask - leader343: received 1 commands after 448.4 ms 09:09:26,328 PM [leader343-writer-152] [DEBUG] JournalWriteTask - leader343: completed 1 commands in 15.22 μs 09:09:26,328 PM [leader343-writer-152] [DEBUG] JournalWriteTask - leader343: journal writer stopped 09:09:26,328 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - leader343: journal closed 09:09:26,328 PM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - leader343: journal closed 09:09:26,328 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader343: stopped executor with 0 remaining tasks 09:09:26,328 PM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - leader343: stopped executor with 0 remaining tasks 09:09:26,328 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$d#-265560408] [INFO] [07/01/2025 21:09:26.330] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 09:09:26,330 PM [test-pekko.actor.default-dispatcher-12] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 09:09:26,332 PM [test-pekko.actor.default-dispatcher-12] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.150:17355] - Exiting completed [WARN] [07/01/2025 21:09:26.332] [test-pekko.actor.default-dispatcher-12] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.170.150:17355) [INFO] [07/01/2025 21:09:26.332] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.150:17355] - Exiting completed [INFO] [07/01/2025 21:09:26.332] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.150:17355] - Shutting down... 09:09:26,332 PM [test-pekko.actor.default-dispatcher-18] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.150:17355] - Shutting down... 09:09:26,334 PM [test-pekko.actor.default-dispatcher-17] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.150:17355] - Successfully shut down [INFO] [07/01/2025 21:09:26.333] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.150:17355] - Successfully shut down [INFO] [07/01/2025 21:09:26.334] [test-pekko.remote.default-remote-dispatcher-10] [pekko://test@10.30.170.150:17355/system/remoting-terminator] Shutting down remote daemon. 09:09:26,334 PM [test-pekko.actor.default-dispatcher-18] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. [INFO] [07/01/2025 21:09:26.334] [test-pekko.remote.default-remote-dispatcher-10] [pekko://test@10.30.170.150:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 09:09:26,334 PM [test-pekko.actor.default-dispatcher-18] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [07/01/2025 21:09:26.336] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.170.150:17355/system/remoting-terminator] Remoting shut down. 09:09:26,337 PM [test-pekko.actor.default-dispatcher-12] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [DEBUG] [07/01/2025 21:09:26.337] [test-pekko.actor.internal-dispatcher-3] [EventStream] shutting down: StandardOutLogger