[DEBUG] [06/30/2025 09:49:40.607] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 09:49:40,607 AM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [06/30/2025 09:49:40.607] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [06/30/2025 09:49:40.607] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [06/30/2025 09:49:40.626] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.12:17355] with UID [5929863146407656568] 09:49:40,626 AM [test-pekko.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.12:17355] with UID [5929863146407656568] [INFO] [06/30/2025 09:49:40.627] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.12:17355] - Starting up, Pekko version [1.0.3] ... 09:49:40,627 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.12:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [06/30/2025 09:49:40.634] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.12:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [06/30/2025 09:49:40.634] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.12:17355] - Started up successfully 09:49:40,634 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.12:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 09:49:40,634 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.12:17355] - Started up successfully 09:49:40,644 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.12:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 09:49:40,644 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.12:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining [INFO] [06/30/2025 09:49:40.644] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.12:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [06/30/2025 09:49:40.644] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.12:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 09:49:40,644 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 09:49:40,644 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:49:40,649 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower279: Starting recovery with journal batch size 1000 09:49:40,649 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: started executor 09:49:40,649 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: started executor 09:49:40,649 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower279: metadata journal open: replayFrom=1 applyTo=0 09:49:40,650 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower279: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:40,650 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower279: journal open: applyTo=0 09:49:40,650 AM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: journal writer started 09:49:40,650 AM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: waiting to receive commands 09:49:40,651 AM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower279] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [06/30/2025 09:49:40.651] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower279]: org.opendaylight.controller.cluster.common.actor.Monitor@1689a96b [INFO] [pekkoDeadLetter][06/30/2025 09:49:40.650] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower279] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [06/30/2025 09:49:40.652] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower280]: org.opendaylight.controller.cluster.common.actor.Monitor@2666af8e [INFO] [pekkoDeadLetter][06/30/2025 09:49:40.652] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower280] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:40,652 AM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower280] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower279/TermInfo.properties does not exist 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower279/journal-v1-1.log 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower279/metadata-v1-1.log 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: no eligible files found 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower279: SnapshotOffer called. 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower279: initializing from snapshot taken at 2025-06-30T09:49:40.645Z 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=1, votedFor=null 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - follower279: Updated server config: isVoting: true, peers: [PeerInfo [id=follower281, address=pekko://test/user/follower281, votingState=NON_VOTING], PeerInfo [id=leader278, address=pekko://test/user/leader278, votingState=VOTING], PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=VOTING]] 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - Recovery snapshot applied for follower279 in 210.8 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower279: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:40.645Z 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower279: support() called 09:49:40,653 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower280: Starting recovery with journal batch size 1000 09:49:40,653 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower279/snapshot-0000000068625DB4-2671EB40.tmp 09:49:40,653 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: started executor 09:49:40,653 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: started executor 09:49:40,654 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower280: metadata journal open: replayFrom=1 applyTo=0 09:49:40,654 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower280: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:40,655 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower280: journal open: applyTo=0 09:49:40,655 AM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: journal writer started 09:49:40,655 AM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: waiting to receive commands 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower280/TermInfo.properties does not exist 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower280/journal-v1-1.log 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower280/metadata-v1-1.log 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: no eligible files found 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower280: SnapshotOffer called. 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower280: initializing from snapshot taken at 2025-06-30T09:49:40.645Z 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=1, votedFor=null 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - follower280: Updated server config: isVoting: true, peers: [PeerInfo [id=follower281, address=pekko://test/user/follower281, votingState=NON_VOTING], PeerInfo [id=leader278, address=pekko://test/user/leader278, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - Recovery snapshot applied for follower280 in 86.82 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower280: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:40.645Z 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower280: support() called 09:49:40,656 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower279/snapshot-0000000068625DB4-2671EB40.v1 09:49:40,656 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower280/snapshot-0000000068625DB4-2671EB40.tmp 09:49:40,657 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower279/journal-v1-1.log 09:49:40,657 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower279/snapshot-0000000068625DB4-2671EB40.v1 09:49:40,657 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower279/metadata-v1-1.log 09:49:40,657 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower279: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:40.645Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower279/snapshot-0000000068625DB4-2671EB40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:40,657 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower279: local snapshot saved in 3.774 ms, deleting Pekko-persisted snapshots [WARN] [06/30/2025 09:49:40.658] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower281]: org.opendaylight.controller.cluster.common.actor.Monitor@28a15532 [INFO] [pekkoDeadLetter][06/30/2025 09:49:40.658] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower281] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:40,658 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower281: Starting recovery with journal batch size 1000 09:49:40,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: started executor 09:49:40,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: started executor 09:49:40,659 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower281: metadata journal open: replayFrom=1 applyTo=0 09:49:40,659 AM [test-pekko.actor.default-dispatcher-15] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower281] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:40,659 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower279: Recovery completed in 6.444 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower281: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower281: journal open: applyTo=0 09:49:40,660 AM [follower281-writer-134] [DEBUG] JournalWriteTask - follower281: journal writer started 09:49:40,660 AM [follower281-writer-134] [DEBUG] JournalWriteTask - follower281: waiting to receive commands 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower281/TermInfo.properties does not exist 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower281/journal-v1-1.log 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower281/metadata-v1-1.log 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: no eligible files found 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower281: SnapshotOffer called. 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower281: initializing from snapshot taken at 2025-06-30T09:49:40.645Z 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower281: Set currentTerm=1, votedFor=null 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PeerInfos - follower281: Updated server config: isVoting: false, peers: [PeerInfo [id=leader278, address=pekko://test/user/leader278, votingState=VOTING], PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - Recovery snapshot applied for follower281 in 38.99 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower281: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:40.645Z 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower281: support() called 09:49:40,660 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower281/snapshot-0000000068625DB4-2671EB40.tmp 09:49:40,661 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower280/snapshot-0000000068625DB4-2671EB40.v1 09:49:40,661 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower280/journal-v1-1.log 09:49:40,661 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower280/snapshot-0000000068625DB4-2671EB40.v1 09:49:40,661 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower280/metadata-v1-1.log 09:49:40,661 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower280: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:40.645Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower280/snapshot-0000000068625DB4-2671EB40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:40,661 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower280: local snapshot saved in 4.595 ms, deleting Pekko-persisted snapshots 09:49:40,661 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower280: Recovery completed in 5.590 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:40,663 AM [test-pekko.actor.default-dispatcher-13] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader278] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:40,663 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=1, votedFor=null 09:49:40,663 AM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - leader278: Starting recovery with journal batch size 1000 09:49:40,663 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower279: Local TermInfo store seeded with TermInfo{term=1} 09:49:40,663 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower279: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6244adf4 09:49:40,663 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower279: Moving last applied index from -1 to -1 09:49:40,663 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: started executor 09:49:40,663 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: started executor 09:49:40,663 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader278: metadata journal open: replayFrom=1 applyTo=0 09:49:40,664 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower281/snapshot-0000000068625DB4-2671EB40.v1 09:49:40,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=1, votedFor=null 09:49:40,664 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower280: Local TermInfo store seeded with TermInfo{term=1} 09:49:40,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower280: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@4f02d782 09:49:40,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower280: Moving last applied index from -1 to -1 09:49:40,664 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower281/journal-v1-1.log 09:49:40,664 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower281/snapshot-0000000068625DB4-2671EB40.v1 09:49:40,664 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower281/metadata-v1-1.log 09:49:40,664 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower281: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:40.645Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/follower281/snapshot-0000000068625DB4-2671EB40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:40,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower280: journal recovery completedin 150.7 μs with journalIndex=1 09:49:40,664 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower281: local snapshot saved in 3.743 ms, deleting Pekko-persisted snapshots 09:49:40,664 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower280: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:40,665 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower281: Recovery completed in 4.389 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader278: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - leader278: journal open: applyTo=0 09:49:40,665 AM [leader278-writer-135] [DEBUG] JournalWriteTask - leader278: journal writer started 09:49:40,665 AM [leader278-writer-135] [DEBUG] JournalWriteTask - leader278: waiting to receive commands 09:49:40,665 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - follower279: journal recovery completedin 1.827 ms with journalIndex=1 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/leader278/TermInfo.properties does not exist 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/leader278/journal-v1-1.log 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/leader278/metadata-v1-1.log 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: no eligible files found 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - leader278: SnapshotOffer called. 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - leader278: initializing from snapshot taken at 2025-06-30T09:49:40.645Z 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader278: Set currentTerm=1, votedFor=null 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - leader278: Updated server config: isVoting: true, peers: [PeerInfo [id=follower281, address=pekko://test/user/follower281, votingState=NON_VOTING], PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for leader278 in 39.01 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - leader278: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:40.645Z 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader278: support() called 09:49:40,665 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/leader278/snapshot-0000000068625DB4-2671EB40.tmp 09:49:40,666 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower279: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) [WARN] [06/30/2025 09:49:40.663] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader278]: org.opendaylight.controller.cluster.common.actor.Monitor@656f7fbd [INFO] [pekkoDeadLetter][06/30/2025 09:49:40.663] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader278] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:40,668 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower281: Set currentTerm=1, votedFor=null 09:49:40,668 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower281: Local TermInfo store seeded with TermInfo{term=1} 09:49:40,668 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower281: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@227f7118 09:49:40,668 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower281: Moving last applied index from -1 to -1 09:49:40,668 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower281: journal recovery completedin 152.1 μs with journalIndex=1 09:49:40,668 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower281: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:40,668 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/leader278/snapshot-0000000068625DB4-2671EB40.v1 09:49:40,668 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/leader278/journal-v1-1.log 09:49:40,668 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/leader278/snapshot-0000000068625DB4-2671EB40.v1 09:49:40,668 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/leader278/metadata-v1-1.log 09:49:40,668 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader278: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:40.645Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15161195255212952835/leader278/snapshot-0000000068625DB4-2671EB40.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:40,668 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - leader278: local snapshot saved in 3.031 ms, deleting Pekko-persisted snapshots 09:49:40,669 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - leader278: Recovery completed in 3.551 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:40,670 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader278: Set currentTerm=1, votedFor=null 09:49:40,670 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - leader278: Local TermInfo store seeded with TermInfo{term=1} 09:49:40,670 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader278: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@3f99bf4b 09:49:40,670 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader278: Moving last applied index from -1 to -1 09:49:40,671 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - leader278: journal recovery completedin 150.0 μs with journalIndex=1 09:49:40,671 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader278: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:41,86 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:49:41,86 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - leader278 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.12:17355, Removed) 09:49:41,86 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - leader278 (Follower): Received ElectionTimeout - switching to Candidate 09:49:41,86 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader278 (Candidate): Election: Candidate has following voting peers: [follower280, follower279] 09:49:41,90 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader278: Set currentTerm=2, votedFor=leader278 09:49:41,90 AM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - leader278 (Candidate): Starting new election term 2 09:49:41,90 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader278 (Candidate): Sending RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} to peer follower280 09:49:41,90 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader278 (Candidate): Sending RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} to peer follower279 09:49:41,91 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - leader278 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:49:41,91 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower280 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:41,91 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower280 (Follower): Cluster state: [] 09:49:41,91 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower280 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:41,91 AM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower280 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:41,91 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower279 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:41,91 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower279 (Follower): Cluster state: [] 09:49:41,91 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower279 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:41,91 AM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - follower279 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:41,94 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=2, votedFor=null 09:49:41,95 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower280 (Follower): In requestVote: RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:41,96 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=2, votedFor=null 09:49:41,96 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower279 (Follower): In requestVote: RequestVote{term=2, candidateId=leader278, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:41,97 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=2, votedFor=leader278 09:49:41,97 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower280 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:41,97 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader278 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:49:41,97 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader278 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:49:41,97 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader278 (Leader): Election: Leader has following peers: [follower281, follower280, follower279] 09:49:41,98 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower281: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:41,98 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower280: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:41,98 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower279: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:41,98 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - leader278 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:49:41,98 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower280: Last sync leader does not match current leader leader278, need to catch up to -1 09:49:41,98 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader278 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower280 updated: matchIndex: -1, nextIndex: 0 09:49:41,98 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower280: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:41,98 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=2, votedFor=leader278 09:49:41,98 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower279 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:41,98 AM [test-pekko.actor.default-dispatcher-15] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:41,99 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower279: Last sync leader does not match current leader leader278, need to catch up to -1 09:49:41,99 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader278 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower279 updated: matchIndex: -1, nextIndex: 0 09:49:41,99 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower279: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:41,99 AM [test-pekko.actor.default-dispatcher-16] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [06/30/2025 09:49:41.099] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower280] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower280, initialSyncDone=false] 09:49:41,99 AM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower281 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader278, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term [INFO] [pekkoDeadLetter][06/30/2025 09:49:41.098] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower280] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [06/30/2025 09:49:41.099] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower279] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower279, initialSyncDone=false] [INFO] [pekkoDeadLetter][06/30/2025 09:49:41.099] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower279] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:41,101 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower281: Set currentTerm=2, votedFor=null 09:49:41,101 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower281: Last sync leader does not match current leader leader278, need to catch up to -1 09:49:41,101 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader278 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower281 updated: matchIndex: -1, nextIndex: 0 09:49:41,102 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower281: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:41.102] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower281] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower281, initialSyncDone=false] [INFO] [pekkoDeadLetter][06/30/2025 09:49:41.102] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower281] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:41,102 AM [test-pekko.actor.default-dispatcher-18] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:41,123 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:49:41,123 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader278 starting 09:49:41,125 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader278: onShutDown 09:49:41,126 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader278: Initiating leader transfer 09:49:41,126 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader278: pauseLeader successfully completed - doing transfer 09:49:41,126 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader278 (Leader): Attempting to transfer leadership 09:49:41,126 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower281: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:41,126 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower281: onLeaderTransitioning: LeaderTransitioning [leaderId=leader278] 09:49:41,126 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower280: onLeaderTransitioning: LeaderTransitioning [leaderId=leader278] 09:49:41,126 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower279: onLeaderTransitioning: LeaderTransitioning [leaderId=leader278] 09:49:41,126 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:49:41,126 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower280: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:41,126 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader278 (Leader): Checking sendAppendEntries for follower follower279: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:41,126 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) [WARN] [06/30/2025 09:49:41.127] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower281] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower281, initialSyncDone=true] [WARN] [06/30/2025 09:49:41.127] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower280] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower280, initialSyncDone=true] 09:49:41,127 AM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:41,127 AM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/30/2025 09:49:41.127] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower281] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower281] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/30/2025 09:49:41.127] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower280] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:41,127 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) [WARN] [06/30/2025 09:49:41.127] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower279] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower279, initialSyncDone=true] 09:49:41,127 AM [test-pekko.actor.default-dispatcher-5] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/30/2025 09:49:41.127] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower279] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 09:49:41,216 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:49:41,216 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:49:41,216 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:49:41,336 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:49:41,336 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:49:41,336 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:49:41,455 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader278: leader transfer aborted 09:49:41,455 AM [test-pekko.actor.default-dispatcher-17] [WARN] RaftActorLeadershipTransferCohort - leader278: Failed to transfer leadership in 329.2 ms 09:49:41,455 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader278: leader transfer failed - sending PoisonPill 09:49:41,455 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader278 (Leader): Leadership transfer expired 09:49:41,455 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:49:41,455 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - leader278: terminating thread VirtualThread[#716,leader278-writer-135]/waiting 09:49:41,455 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:49:41,455 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower281: Lagging 0 entries behind leader leader278 and reached -1 (of expected -1) 09:49:41,456 AM [leader278-writer-135] [DEBUG] JournalWriteTask - leader278: received 1 commands after 790.7 ms 09:49:41,456 AM [leader278-writer-135] [DEBUG] JournalWriteTask - leader278: completed 1 commands in 31.47 μs 09:49:41,456 AM [leader278-writer-135] [DEBUG] JournalWriteTask - leader278: journal writer stopped 09:49:41,456 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader278: journal closed 09:49:41,456 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader278: journal closed 09:49:41,456 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: stopped executor with 0 remaining tasks 09:49:41,456 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader278: stopped executor with 0 remaining tasks [WARN] [06/30/2025 09:49:41.456] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader278] received dead letter from TestActor[pekko://test/user/follower280]: AppendEntriesReply{term=2, success=true, followerId=follower280, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} [WARN] [06/30/2025 09:49:41.456] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader278] received dead letter from TestActor[pekko://test/user/follower279]: AppendEntriesReply{term=2, success=true, followerId=follower279, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} [WARN] [06/30/2025 09:49:41.456] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader278] received dead letter from TestActor[pekko://test/user/follower281]: AppendEntriesReply{term=2, success=true, followerId=follower281, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:49:41,457 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader278 ending 09:49:41,459 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 09:49:41,459 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower279-notifier282#-463695742] 09:49:41,470 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower279-collector283#-640502555] 09:49:41,470 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower279] 09:49:41,471 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower279: terminating thread VirtualThread[#707,follower279-writer-132]/waiting 09:49:41,471 AM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: received 1 commands after 820.7 ms 09:49:41,471 AM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: completed 1 commands in 27.86 μs 09:49:41,471 AM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: journal writer stopped 09:49:41,471 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower279: journal closed 09:49:41,471 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower279: journal closed 09:49:41,471 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower279: stopped executor with 0 remaining tasks 09:49:41,471 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower279: stopped executor with 0 remaining tasks 09:49:41,471 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower280-notifier284#-790535274] 09:49:41,472 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower280-collector285#667478787] 09:49:41,472 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower280] 09:49:41,472 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower280: terminating thread VirtualThread[#710,follower280-writer-133]/waiting 09:49:41,472 AM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: received 1 commands after 817.2 ms 09:49:41,472 AM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: completed 1 commands in 17.37 μs 09:49:41,472 AM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: journal writer stopped 09:49:41,474 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower280: journal closed 09:49:41,474 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower280: journal closed 09:49:41,474 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower280: stopped executor with 0 remaining tasks 09:49:41,474 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower280: stopped executor with 0 remaining tasks 09:49:41,475 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower281-notifier286#-1844297469] 09:49:41,475 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower281-collector287#1080938798] 09:49:41,475 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower281] 09:49:41,475 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower281: terminating thread VirtualThread[#714,follower281-writer-134]/waiting 09:49:41,475 AM [follower281-writer-134] [DEBUG] JournalWriteTask - follower281: received 1 commands after 815.6 ms 09:49:41,475 AM [follower281-writer-134] [DEBUG] JournalWriteTask - follower281: completed 1 commands in 16.94 μs 09:49:41,475 AM [follower281-writer-134] [DEBUG] JournalWriteTask - follower281: journal writer stopped 09:49:41,476 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower281: journal closed 09:49:41,476 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower281: journal closed 09:49:41,476 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower281: stopped executor with 0 remaining tasks 09:49:41,476 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower281: stopped executor with 0 remaining tasks 09:49:41,476 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader278-notifier288#-135017737] 09:49:41,476 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader278-collector289#1850334766] 09:49:41,476 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader278] 09:49:41,479 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 09:49:41,479 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/30/2025 09:49:41.481] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower291]: org.opendaylight.controller.cluster.common.actor.Monitor@e7b93b1 09:49:41,482 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActor - follower291: Starting recovery with journal batch size 1000 09:49:41,483 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: started executor 09:49:41,483 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: started executor 09:49:41,483 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower291: metadata journal open: replayFrom=1 applyTo=0 09:49:41,484 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower291: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:41,484 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower291: journal open: applyTo=0 09:49:41,484 AM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: journal writer started 09:49:41,484 AM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: waiting to receive commands 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower291/TermInfo.properties does not exist 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower291/journal-v1-1.log 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower291/metadata-v1-1.log 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: no eligible files found 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower291: SnapshotOffer called. 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower291: initializing from snapshot taken at 2025-06-30T09:49:41.479Z 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=1, votedFor=null 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - follower291: Updated server config: isVoting: true, peers: [PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=VOTING], PeerInfo [id=follower293, address=pekko://test/user/follower293, votingState=NON_VOTING], PeerInfo [id=leader290, address=pekko://test/user/leader290, votingState=VOTING]] 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for follower291 in 80.82 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower291: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:41.479Z 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - follower291: support() called 09:49:41,485 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower291/snapshot-0000000068625DB5-1C8CF5C0.tmp 09:49:41,491 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower291/snapshot-0000000068625DB5-1C8CF5C0.v1 09:49:41,491 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower291/snapshot-0000000068625DB5-1C8CF5C0.v1 09:49:41,491 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower291/journal-v1-1.log 09:49:41,491 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower291/metadata-v1-1.log 09:49:41,491 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:41.479Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower291/snapshot-0000000068625DB5-1C8CF5C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:41,491 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower291: local snapshot saved in 6.179 ms, deleting Pekko-persisted snapshots 09:49:41,492 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower291: Recovery completed in 7.281 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:41,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=1, votedFor=null 09:49:41,494 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower291: Local TermInfo store seeded with TermInfo{term=1} 09:49:41,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower291: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@31509a42 09:49:41,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractReplicatedLog - follower291: Moving last applied index from -1 to -1 09:49:41,495 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] JournalRecovery - follower291: journal recovery completedin 231.9 μs with journalIndex=1 09:49:41,495 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower291: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) [WARN] [06/30/2025 09:49:41.583] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower292]: org.opendaylight.controller.cluster.common.actor.Monitor@468b8160 09:49:41,585 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - follower292: Starting recovery with journal batch size 1000 09:49:41,585 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: started executor 09:49:41,585 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: started executor 09:49:41,585 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower292: metadata journal open: replayFrom=1 applyTo=0 09:49:41,586 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower292: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:41,586 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower292: journal open: applyTo=0 09:49:41,587 AM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: journal writer started 09:49:41,587 AM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: waiting to receive commands 09:49:41,587 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower292/TermInfo.properties does not exist 09:49:41,587 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower292/journal-v1-1.log 09:49:41,587 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower292/metadata-v1-1.log 09:49:41,587 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: no eligible files found 09:49:41,587 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower292: SnapshotOffer called. 09:49:41,587 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower292: initializing from snapshot taken at 2025-06-30T09:49:41.479Z 09:49:41,587 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=1, votedFor=null 09:49:41,587 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower292: Updated server config: isVoting: true, peers: [PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=follower293, address=pekko://test/user/follower293, votingState=NON_VOTING], PeerInfo [id=leader290, address=pekko://test/user/leader290, votingState=VOTING]] 09:49:41,588 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower292 in 69.11 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:41,588 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower292: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:41.479Z 09:49:41,588 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower292: support() called 09:49:41,588 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower292/snapshot-0000000068625DB5-1C8CF5C0.tmp [WARN] [06/30/2025 09:49:41.588] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower293]: org.opendaylight.controller.cluster.common.actor.Monitor@4f991df4 09:49:41,589 AM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower293: Starting recovery with journal batch size 1000 09:49:41,589 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower293: started executor 09:49:41,590 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower293: started executor 09:49:41,590 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower293: metadata journal open: replayFrom=1 applyTo=0 09:49:41,591 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower292/snapshot-0000000068625DB5-1C8CF5C0.v1 09:49:41,591 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower293: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:41,591 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower293: journal open: applyTo=0 09:49:41,591 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower292/snapshot-0000000068625DB5-1C8CF5C0.v1 09:49:41,591 AM [follower293-writer-138] [DEBUG] JournalWriteTask - follower293: journal writer started 09:49:41,591 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower292/journal-v1-1.log 09:49:41,591 AM [follower293-writer-138] [DEBUG] JournalWriteTask - follower293: waiting to receive commands 09:49:41,591 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower292/metadata-v1-1.log 09:49:41,591 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower292: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:41.479Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower292/snapshot-0000000068625DB5-1C8CF5C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:41,591 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower292: local snapshot saved in 3.384 ms, deleting Pekko-persisted snapshots 09:49:41,592 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower292: Recovery completed in 4.419 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:41,592 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower293/TermInfo.properties does not exist 09:49:41,593 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower293/journal-v1-1.log 09:49:41,593 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower293/metadata-v1-1.log 09:49:41,593 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower293: no eligible files found 09:49:41,593 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower293: SnapshotOffer called. 09:49:41,593 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower293: initializing from snapshot taken at 2025-06-30T09:49:41.479Z 09:49:41,593 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=1, votedFor=null 09:49:41,593 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - follower293: Updated server config: isVoting: false, peers: [PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=VOTING], PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=leader290, address=pekko://test/user/leader290, votingState=VOTING]] 09:49:41,593 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for follower293 in 113.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:41,593 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower293: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:41.479Z 09:49:41,593 AM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - follower293: support() called 09:49:41,594 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower293: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower293/snapshot-0000000068625DB5-1C8CF5C0.tmp [WARN] [06/30/2025 09:49:41.594] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader290]: org.opendaylight.controller.cluster.common.actor.Monitor@794ec531 09:49:41,594 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - leader290: Starting recovery with journal batch size 1000 09:49:41,594 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: started executor 09:49:41,594 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: started executor 09:49:41,595 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=1, votedFor=null 09:49:41,595 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower292: Local TermInfo store seeded with TermInfo{term=1} 09:49:41,595 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower292: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@90558f6 09:49:41,595 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower292: Moving last applied index from -1 to -1 09:49:41,595 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader290: metadata journal open: replayFrom=1 applyTo=0 09:49:41,595 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower292: journal recovery completedin 289.1 μs with journalIndex=1 09:49:41,596 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader290: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:41,596 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader290: journal open: applyTo=0 09:49:41,596 AM [leader290-writer-139] [DEBUG] JournalWriteTask - leader290: journal writer started 09:49:41,596 AM [leader290-writer-139] [DEBUG] JournalWriteTask - leader290: waiting to receive commands 09:49:41,596 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower292: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:41,596 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower293: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower293/snapshot-0000000068625DB5-1C8CF5C0.v1 09:49:41,596 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower293: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower293/snapshot-0000000068625DB5-1C8CF5C0.v1 09:49:41,596 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower293/journal-v1-1.log 09:49:41,596 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower293: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower293/metadata-v1-1.log 09:49:41,596 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/leader290/TermInfo.properties does not exist 09:49:41,596 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower293: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:41.479Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/follower293/snapshot-0000000068625DB5-1C8CF5C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:41,596 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/leader290/journal-v1-1.log 09:49:41,596 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/leader290/metadata-v1-1.log 09:49:41,596 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower293: local snapshot saved in 3.185 ms, deleting Pekko-persisted snapshots 09:49:41,596 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: no eligible files found 09:49:41,596 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - leader290: SnapshotOffer called. 09:49:41,596 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - leader290: initializing from snapshot taken at 2025-06-30T09:49:41.479Z 09:49:41,597 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=1, votedFor=null 09:49:41,597 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - leader290: Updated server config: isVoting: true, peers: [PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=VOTING], PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=follower293, address=pekko://test/user/follower293, votingState=NON_VOTING]] 09:49:41,597 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for leader290 in 51.60 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:41,597 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader290: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:41.479Z 09:49:41,597 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader290: support() called 09:49:41,597 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/leader290/snapshot-0000000068625DB5-1C8CF5C0.tmp 09:49:41,597 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower293: Recovery completed in 4.228 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:41,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/leader290/snapshot-0000000068625DB5-1C8CF5C0.v1 09:49:41,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/leader290/snapshot-0000000068625DB5-1C8CF5C0.v1 09:49:41,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/leader290/journal-v1-1.log 09:49:41,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/leader290/metadata-v1-1.log 09:49:41,600 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader290: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:41.479Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15679394977610426245/leader290/snapshot-0000000068625DB5-1C8CF5C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:41,600 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader290: local snapshot saved in 3.149 ms, deleting Pekko-persisted snapshots 09:49:41,601 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader290: Recovery completed in 4.030 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:41,601 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=1, votedFor=null 09:49:41,601 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower293: Local TermInfo store seeded with TermInfo{term=1} 09:49:41,601 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower293: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@652e71db 09:49:41,601 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower293: Moving last applied index from -1 to -1 09:49:41,601 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower293: journal recovery completedin 210.2 μs with journalIndex=1 09:49:41,601 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower293: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:41,602 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=1, votedFor=null 09:49:41,602 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader290: Local TermInfo store seeded with TermInfo{term=1} 09:49:41,602 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader290: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@65d09e93 09:49:41,602 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader290: Moving last applied index from -1 to -1 09:49:41,603 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - leader290: journal recovery completedin 151.3 μs with journalIndex=1 09:49:41,603 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader290: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:41,995 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:49:41,995 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - leader290 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.12:17355, Removed) 09:49:41,995 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - leader290 (Follower): Received ElectionTimeout - switching to Candidate 09:49:41,995 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader290 (Candidate): Election: Candidate has following voting peers: [follower292, follower291] 09:49:42,0 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=2, votedFor=leader290 09:49:42,0 AM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - leader290 (Candidate): Starting new election term 2 09:49:42,0 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader290 (Candidate): Sending RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} to peer follower292 09:49:42,0 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader290 (Candidate): Sending RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} to peer follower291 09:49:42,0 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower292 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:42,0 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - leader290 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:49:42,0 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower292 (Follower): Cluster state: [] 09:49:42,0 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower291 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:42,0 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower292 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:42,0 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower292 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:42,0 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower291 (Follower): Cluster state: [] 09:49:42,0 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower291 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:42,0 AM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower291 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:42,3 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=2, votedFor=null 09:49:42,3 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower291 (Follower): In requestVote: RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:42,5 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=2, votedFor=null 09:49:42,5 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower292 (Follower): In requestVote: RequestVote{term=2, candidateId=leader290, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:42,7 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=2, votedFor=leader290 09:49:42,7 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower291 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:42,7 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader290 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:49:42,7 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader290 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:49:42,7 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader290 (Leader): Election: Leader has following peers: [follower292, follower291, follower293] 09:49:42,7 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,7 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,7 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,7 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader290 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:49:42,7 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower291: Last sync leader does not match current leader leader290, need to catch up to -1 09:49:42,7 AM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - follower293 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader290, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 09:49:42,7 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader290 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower291 updated: matchIndex: -1, nextIndex: 0 09:49:42,8 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:42.008] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=false] 09:49:42,9 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=2, votedFor=leader290 09:49:42,9 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower292 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:42,9 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower292: Last sync leader does not match current leader leader290, need to catch up to -1 09:49:42,9 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader290 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower292 updated: matchIndex: -1, nextIndex: 0 [WARN] [06/30/2025 09:49:42.009] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower292] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower292, initialSyncDone=false] 09:49:42,10 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,10 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=2, votedFor=null 09:49:42,10 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower293: Last sync leader does not match current leader leader290, need to catch up to -1 09:49:42,10 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader290 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower293 updated: matchIndex: -1, nextIndex: 0 09:49:42,10 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:42.010] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower293] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower293, initialSyncDone=false] 09:49:42,56 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:49:42,56 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 09:49:42,56 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 09:49:42,56 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 09:49:42,57 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 09:49:42,57 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader290: onRequestLeadership RequestLeadership [requestedFollowerId=follower292, replyTo=Actor[pekko://test/user/$a#-756740832]] 09:49:42,58 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader290: Initiating leader transfer 09:49:42,59 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower292: onLeaderTransitioning: LeaderTransitioning [leaderId=leader290] 09:49:42,59 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower291: onLeaderTransitioning: LeaderTransitioning [leaderId=leader290] 09:49:42,59 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader290: pauseLeader successfully completed - doing transfer 09:49:42,59 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower293: onLeaderTransitioning: LeaderTransitioning [leaderId=leader290] 09:49:42,59 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader290 (Leader): Attempting to transfer leadership 09:49:42,59 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,59 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,59 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,59 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower292: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 09:49:42,59 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower291: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 09:49:42,59 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower293: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) [WARN] [06/30/2025 09:49:42.059] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower292] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower292, initialSyncDone=true] [WARN] [06/30/2025 09:49:42.059] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower293] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower293, initialSyncDone=true] [WARN] [06/30/2025 09:49:42.059] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=true] 09:49:42,60 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader290 (Leader): tryToCompleteLeadershipTransfer: followerId: follower292, matchIndex: -1, lastIndex: -1, isVoting: true 09:49:42,60 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader290 (Leader): Follower's log matches - sending ElectionTimeout 09:49:42,60 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,60 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,60 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader290 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,60 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower292: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 09:49:42,60 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower291: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 09:49:42,60 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower293: Lagging 0 entries behind leader leader290 and reached -1 (of expected -1) 09:49:42,60 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:49:42,60 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower292 (Follower): Received TimeoutNow - switching to Candidate 09:49:42,60 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - follower292 (Candidate): Election: Candidate has following voting peers: [follower291, leader290] 09:49:42,61 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader290 (Leader): Leader transfer complete 09:49:42,61 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader290: leader transfer complete - waiting for new leader 09:49:42,63 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=3, votedFor=follower292 09:49:42,63 AM [test-pekko.actor.default-dispatcher-13] [INFO] Candidate - follower292 (Candidate): Starting new election term 3 09:49:42,63 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - follower292 (Candidate): Sending RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} to peer follower291 09:49:42,63 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - follower292 (Candidate): Sending RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} to peer leader290 09:49:42,63 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - follower292 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 09:49:42,63 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower291 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:42,63 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower291 (Follower): Cluster state: [] 09:49:42,63 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower291 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:42,63 AM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower291 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 2 - updating term 09:49:42,63 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader290 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:42,63 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader290 (Leader): Cluster state: [] 09:49:42,63 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader290 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:42,63 AM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractLeader - leader290 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower 09:49:42,66 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=3, votedFor=null 09:49:42,66 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader290 (Leader): Leadership transfer in progress - processing RequestVote 09:49:42,66 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader290 (Leader): In requestVote: RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:42,68 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=3, votedFor=null 09:49:42,68 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower291 (Follower): In requestVote: RequestVote{term=3, candidateId=follower292, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:42,69 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader290: Set currentTerm=3, votedFor=follower292 09:49:42,69 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader290 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 09:49:42,69 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - follower292 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 09:49:42,69 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - follower292 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 09:49:42,69 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - leader290 (Leader) :- Switching from behavior Leader to Follower, election term: 3 09:49:42,69 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): Election: Leader has following peers: [follower291, follower293, leader290] 09:49:42,69 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower291: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,69 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower293: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,69 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower leader290: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,69 AM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - follower292 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 09:49:42,69 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower293 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower292, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 09:49:42,69 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - leader290: Last sync leader does not match current leader follower292, need to catch up to -1 09:49:42,69 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader290: leader changed to follower292 09:49:42,69 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorLeadershipTransferCohort - leader290: Successfully transferred leadership to follower292 in 10.65 ms 09:49:42,69 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader290 updated: matchIndex: -1, nextIndex: 0 09:49:42,69 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader290: Leadership transferred successfully to follower292 09:49:42,69 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower leader290: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:42.069] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader290] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader290, initialSyncDone=false] 09:49:42,70 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=3, votedFor=follower292 09:49:42,70 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower291 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 09:49:42,70 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower291: Last sync leader does not match current leader follower292, need to catch up to -1 09:49:42,70 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower291 updated: matchIndex: -1, nextIndex: 0 09:49:42,70 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:42.070] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=false] 09:49:42,71 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower293: Set currentTerm=3, votedFor=null 09:49:42,71 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower293: Last sync leader does not match current leader follower292, need to catch up to -1 09:49:42,71 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower292 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower293 updated: matchIndex: -1, nextIndex: 0 09:49:42,71 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower292 (Leader): Checking sendAppendEntries for follower follower293: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:42.071] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower293] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower293, initialSyncDone=false] 09:49:42,108 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 09:49:42,108 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower291-notifier294#1067621262] 09:49:42,109 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower291-collector295#2001925934] 09:49:42,110 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower291] 09:49:42,110 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower291: terminating thread VirtualThread[#717,follower291-writer-136]/waiting 09:49:42,110 AM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: received 1 commands after 626.0 ms 09:49:42,110 AM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: completed 1 commands in 25.12 μs 09:49:42,110 AM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: journal writer stopped 09:49:42,110 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower291: journal closed 09:49:42,110 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower291: journal closed 09:49:42,110 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower291: stopped executor with 0 remaining tasks 09:49:42,110 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower291: stopped executor with 0 remaining tasks 09:49:42,110 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower292-notifier296#-1124202186] 09:49:42,111 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower292-collector297#-1750344094] 09:49:42,111 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower292] 09:49:42,111 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower292: terminating thread VirtualThread[#718,follower292-writer-137]/waiting 09:49:42,111 AM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: received 1 commands after 524.5 ms 09:49:42,111 AM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: completed 1 commands in 7.240 μs 09:49:42,111 AM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: journal writer stopped 09:49:42,111 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower292: journal closed 09:49:42,111 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower292: journal closed 09:49:42,111 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower292: stopped executor with 0 remaining tasks 09:49:42,111 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower292: stopped executor with 0 remaining tasks 09:49:42,111 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower293-notifier298#-505245720] 09:49:42,111 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower293-collector299#14025137] 09:49:42,112 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower293] 09:49:42,112 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower293: terminating thread VirtualThread[#719,follower293-writer-138]/waiting 09:49:42,112 AM [follower293-writer-138] [DEBUG] JournalWriteTask - follower293: received 1 commands after 520.9 ms 09:49:42,112 AM [follower293-writer-138] [DEBUG] JournalWriteTask - follower293: completed 1 commands in 10.97 μs 09:49:42,112 AM [follower293-writer-138] [DEBUG] JournalWriteTask - follower293: journal writer stopped 09:49:42,112 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower293: journal closed 09:49:42,112 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower293: journal closed 09:49:42,112 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower293: stopped executor with 0 remaining tasks 09:49:42,112 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower293: stopped executor with 0 remaining tasks 09:49:42,112 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader290-notifier300#-1465390873] 09:49:42,112 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader290-collector301#1026679640] 09:49:42,112 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader290] 09:49:42,113 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - leader290: terminating thread VirtualThread[#720,leader290-writer-139]/waiting 09:49:42,113 AM [leader290-writer-139] [DEBUG] JournalWriteTask - leader290: received 1 commands after 517.0 ms 09:49:42,113 AM [leader290-writer-139] [DEBUG] JournalWriteTask - leader290: completed 1 commands in 10.79 μs 09:49:42,113 AM [leader290-writer-139] [DEBUG] JournalWriteTask - leader290: journal writer stopped 09:49:42,113 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader290: journal closed 09:49:42,113 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - leader290: journal closed 09:49:42,113 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader290: stopped executor with 0 remaining tasks 09:49:42,113 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader290: stopped executor with 0 remaining tasks 09:49:42,113 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$a#-756740832] 09:49:42,116 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 09:49:42,116 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/30/2025 09:49:42.119] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower303]: org.opendaylight.controller.cluster.common.actor.Monitor@20b4a0f1 09:49:42,119 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower303: Starting recovery with journal batch size 1000 09:49:42,119 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: started executor 09:49:42,120 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: started executor 09:49:42,120 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower303: metadata journal open: replayFrom=1 applyTo=0 09:49:42,120 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower303: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:42,120 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower303: journal open: applyTo=0 09:49:42,120 AM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: journal writer started 09:49:42,120 AM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: waiting to receive commands 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower303/TermInfo.properties does not exist 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower303/journal-v1-1.log 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower303/metadata-v1-1.log 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower303: no eligible files found 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower303: SnapshotOffer called. 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower303: initializing from snapshot taken at 2025-06-30T09:49:42.116Z 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=1, votedFor=null 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - follower303: Updated server config: isVoting: true, peers: [PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=VOTING], PeerInfo [id=leader302, address=pekko://test/user/leader302, votingState=VOTING], PeerInfo [id=follower305, address=pekko://test/user/follower305, votingState=NON_VOTING]] 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - Recovery snapshot applied for follower303 in 72.76 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower303: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:42.116Z 09:49:42,121 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower303: support() called 09:49:42,122 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower303: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower303/snapshot-0000000068625DB6-06EA0500.tmp [WARN] [06/30/2025 09:49:42.122] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower304]: org.opendaylight.controller.cluster.common.actor.Monitor@1fec671a 09:49:42,123 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower304: Starting recovery with journal batch size 1000 09:49:42,123 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: started executor 09:49:42,123 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower304: started executor 09:49:42,123 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower304: metadata journal open: replayFrom=1 applyTo=0 09:49:42,124 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower303: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower303/snapshot-0000000068625DB6-06EA0500.v1 09:49:42,124 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower303: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower303/snapshot-0000000068625DB6-06EA0500.v1 09:49:42,124 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower303/journal-v1-1.log 09:49:42,124 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower303/metadata-v1-1.log 09:49:42,124 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower303: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:42.116Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower303/snapshot-0000000068625DB6-06EA0500.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:42,124 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower303: local snapshot saved in 2.539 ms, deleting Pekko-persisted snapshots 09:49:42,124 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower304: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:42,124 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower304: journal open: applyTo=0 09:49:42,124 AM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: journal writer started 09:49:42,124 AM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: waiting to receive commands 09:49:42,125 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower303: Recovery completed in 3.601 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower304/TermInfo.properties does not exist 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower304/journal-v1-1.log 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower304/metadata-v1-1.log 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: no eligible files found 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower304: SnapshotOffer called. 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower304: initializing from snapshot taken at 2025-06-30T09:49:42.116Z 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=1, votedFor=null 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower304: Updated server config: isVoting: true, peers: [PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=leader302, address=pekko://test/user/leader302, votingState=VOTING], PeerInfo [id=follower305, address=pekko://test/user/follower305, votingState=NON_VOTING]] 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower304 in 86.45 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower304: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:42.116Z 09:49:42,125 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower304: support() called 09:49:42,126 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower304/snapshot-0000000068625DB6-06EA0500.tmp [WARN] [06/30/2025 09:49:42.126] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower305]: org.opendaylight.controller.cluster.common.actor.Monitor@19d8ffa1 09:49:42,126 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower305: Starting recovery with journal batch size 1000 09:49:42,127 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: started executor 09:49:42,127 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: started executor 09:49:42,127 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower305: metadata journal open: replayFrom=1 applyTo=0 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower305: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower305: journal open: applyTo=0 09:49:42,128 AM [follower305-writer-142] [DEBUG] JournalWriteTask - follower305: journal writer started 09:49:42,128 AM [follower305-writer-142] [DEBUG] JournalWriteTask - follower305: waiting to receive commands 09:49:42,128 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower304/snapshot-0000000068625DB6-06EA0500.v1 09:49:42,128 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower304/snapshot-0000000068625DB6-06EA0500.v1 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower305/TermInfo.properties does not exist 09:49:42,128 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower304/journal-v1-1.log 09:49:42,128 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower304/metadata-v1-1.log 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower305/journal-v1-1.log 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower305/metadata-v1-1.log 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: no eligible files found 09:49:42,128 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:42.116Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower304/snapshot-0000000068625DB6-06EA0500.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower305: SnapshotOffer called. 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower305: initializing from snapshot taken at 2025-06-30T09:49:42.116Z 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower305: Set currentTerm=1, votedFor=null 09:49:42,128 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower304: local snapshot saved in 2.874 ms, deleting Pekko-persisted snapshots 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - follower305: Updated server config: isVoting: false, peers: [PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=VOTING], PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=leader302, address=pekko://test/user/leader302, votingState=VOTING]] 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - Recovery snapshot applied for follower305 in 76.54 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower305: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:42.116Z 09:49:42,128 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower305: support() called 09:49:42,129 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=1, votedFor=null 09:49:42,129 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower303: Local TermInfo store seeded with TermInfo{term=1} 09:49:42,129 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower303: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@676af468 09:49:42,129 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower303: Moving last applied index from -1 to -1 09:49:42,129 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower305/snapshot-0000000068625DB6-06EA0500.tmp 09:49:42,129 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower303: journal recovery completedin 222.6 μs with journalIndex=1 09:49:42,129 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower303: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:42,129 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower304: Recovery completed in 4.156 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [06/30/2025 09:49:42.129] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader302]: org.opendaylight.controller.cluster.common.actor.Monitor@6545735a 09:49:42,130 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader302: Starting recovery with journal batch size 1000 09:49:42,130 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: started executor 09:49:42,130 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: started executor 09:49:42,130 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader302: metadata journal open: replayFrom=1 applyTo=0 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader302: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRaftStorage - leader302: journal open: applyTo=0 09:49:42,131 AM [leader302-writer-143] [DEBUG] JournalWriteTask - leader302: journal writer started 09:49:42,131 AM [leader302-writer-143] [DEBUG] JournalWriteTask - leader302: waiting to receive commands 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/leader302/TermInfo.properties does not exist 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/leader302/journal-v1-1.log 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/leader302/metadata-v1-1.log 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: no eligible files found 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PekkoRecovery - leader302: SnapshotOffer called. 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PekkoRecovery - leader302: initializing from snapshot taken at 2025-06-30T09:49:42.116Z 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader302: Set currentTerm=1, votedFor=null 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - leader302: Updated server config: isVoting: true, peers: [PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=VOTING], PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=follower305, address=pekko://test/user/follower305, votingState=NON_VOTING]] 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - Recovery snapshot applied for leader302 in 49.23 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - leader302: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:42.116Z 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - leader302: support() called 09:49:42,131 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/leader302/snapshot-0000000068625DB6-06EA0500.tmp 09:49:42,132 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower305/snapshot-0000000068625DB6-06EA0500.v1 09:49:42,132 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower305/snapshot-0000000068625DB6-06EA0500.v1 09:49:42,132 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower305/journal-v1-1.log 09:49:42,132 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower305/metadata-v1-1.log 09:49:42,132 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:42.116Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/follower305/snapshot-0000000068625DB6-06EA0500.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:42,132 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower305: local snapshot saved in 3.315 ms, deleting Pekko-persisted snapshots 09:49:42,132 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower305: Recovery completed in 3.964 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:42,133 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=1, votedFor=null 09:49:42,133 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower304: Local TermInfo store seeded with TermInfo{term=1} 09:49:42,133 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower304: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@59f73f92 09:49:42,133 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower304: Moving last applied index from -1 to -1 09:49:42,133 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower304: journal recovery completedin 218.0 μs with journalIndex=1 09:49:42,134 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower304: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:42,134 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/leader302/snapshot-0000000068625DB6-06EA0500.v1 09:49:42,134 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/leader302/snapshot-0000000068625DB6-06EA0500.v1 09:49:42,134 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/leader302/journal-v1-1.log 09:49:42,134 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/leader302/metadata-v1-1.log 09:49:42,134 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader302: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:42.116Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8266039311195059399/leader302/snapshot-0000000068625DB6-06EA0500.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:42,135 AM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - leader302: local snapshot saved in 3.225 ms, deleting Pekko-persisted snapshots 09:49:42,135 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower305: Set currentTerm=1, votedFor=null 09:49:42,135 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower305: Local TermInfo store seeded with TermInfo{term=1} 09:49:42,135 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower305: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@b182ff0 09:49:42,135 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractReplicatedLog - follower305: Moving last applied index from -1 to -1 09:49:42,135 AM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - leader302: Recovery completed in 3.898 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:42,135 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] JournalRecovery - follower305: journal recovery completedin 191.7 μs with journalIndex=1 09:49:42,136 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower305: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:42,137 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader302: Set currentTerm=1, votedFor=null 09:49:42,137 AM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - leader302: Local TermInfo store seeded with TermInfo{term=1} 09:49:42,137 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader302: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@779c96c1 09:49:42,137 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader302: Moving last applied index from -1 to -1 09:49:42,137 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - leader302: journal recovery completedin 159.9 μs with journalIndex=1 09:49:42,138 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader302: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:42,555 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:49:42,555 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - leader302 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.12:17355, Removed) 09:49:42,555 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - leader302 (Follower): Received ElectionTimeout - switching to Candidate 09:49:42,555 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader302 (Candidate): Election: Candidate has following voting peers: [follower304, follower303] 09:49:42,558 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader302: Set currentTerm=2, votedFor=leader302 09:49:42,558 AM [test-pekko.actor.default-dispatcher-13] [INFO] Candidate - leader302 (Candidate): Starting new election term 2 09:49:42,559 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader302 (Candidate): Sending RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} to peer follower304 09:49:42,559 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader302 (Candidate): Sending RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} to peer follower303 09:49:42,559 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - leader302 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:49:42,559 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower304 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:42,559 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower304 (Follower): Cluster state: [] 09:49:42,559 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower304 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:42,559 AM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - follower304 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:42,559 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower303 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:42,559 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower303 (Follower): Cluster state: [] 09:49:42,559 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower303 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:42,559 AM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower303 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:42,562 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=2, votedFor=null 09:49:42,562 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower304 (Follower): In requestVote: RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:42,563 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=2, votedFor=null 09:49:42,564 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower303 (Follower): In requestVote: RequestVote{term=2, candidateId=leader302, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:42,565 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=2, votedFor=leader302 09:49:42,565 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower304 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:42,566 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader302 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:49:42,566 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader302 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:49:42,566 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Election: Leader has following peers: [follower304, follower303, follower305] 09:49:42,566 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower304: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,566 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower303: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,566 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower305: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,566 AM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower305 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 09:49:42,566 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower304: Last sync leader does not match current leader leader302, need to catch up to -1 09:49:42,566 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=2, votedFor=leader302 09:49:42,566 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower303 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:42,566 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower303: Last sync leader does not match current leader leader302, need to catch up to -1 [WARN] [06/30/2025 09:49:42.566] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower304] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower304, initialSyncDone=false] 09:49:42,566 AM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader302 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 [WARN] [06/30/2025 09:49:42.566] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower303] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower303, initialSyncDone=false] 09:49:42,567 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower304 updated: matchIndex: -1, nextIndex: 0 09:49:42,567 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower304: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,567 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower303 updated: matchIndex: -1, nextIndex: 0 09:49:42,567 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower303: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,569 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower305: Set currentTerm=2, votedFor=null 09:49:42,569 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower305: Last sync leader does not match current leader leader302, need to catch up to -1 09:49:42,569 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower305 updated: matchIndex: -1, nextIndex: 0 09:49:42,569 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower305: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:42.569] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower305] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower305, initialSyncDone=false] 09:49:42,590 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:49:42,590 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 09:49:42,591 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 09:49:42,591 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower304 starting 09:49:42,591 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower304: onShutDown 09:49:42,591 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower304: terminating thread VirtualThread[#722,follower304-writer-141]/waiting 09:49:42,591 AM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: received 1 commands after 467.0 ms 09:49:42,592 AM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: completed 1 commands in 30.33 μs 09:49:42,592 AM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: journal writer stopped 09:49:42,592 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower304: journal closed 09:49:42,592 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower304: journal closed 09:49:42,592 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower304: stopped executor with 0 remaining tasks 09:49:42,592 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower304: stopped executor with 0 remaining tasks 09:49:42,592 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower304 ending 09:49:42,592 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 09:49:42,592 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 09:49:42,592 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader302: onRequestLeadership RequestLeadership [requestedFollowerId=follower304, replyTo=Actor[pekko://test/user/$b#63188880]] 09:49:42,592 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader302: Initiating leader transfer 09:49:42,592 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader302: pauseLeader successfully completed - doing transfer 09:49:42,592 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Leader - leader302 (Leader): Attempting to transfer leadership [WARN] [06/30/2025 09:49:42.592] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: LeaderTransitioning [leaderId=leader302] 09:49:42,592 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower303: onLeaderTransitioning: LeaderTransitioning [leaderId=leader302] 09:49:42,592 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower305: onLeaderTransitioning: LeaderTransitioning [leaderId=leader302] 09:49:42,592 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower304: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,593 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower303: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:42,593 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader302 (Leader): Checking sendAppendEntries for follower follower305: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:42.593] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:49:42,593 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower303: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 09:49:42,593 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower305: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) [WARN] [06/30/2025 09:49:42.593] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower303] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower303, initialSyncDone=true] [WARN] [06/30/2025 09:49:42.593] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower305] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower305, initialSyncDone=true] [WARN] [06/30/2025 09:49:42.686] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [06/30/2025 09:49:42.805] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:49:42,805 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower303: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 09:49:42,805 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower305: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 09:49:42,925 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader302: leader transfer aborted 09:49:42,925 AM [test-pekko.actor.default-dispatcher-11] [WARN] RaftActorLeadershipTransferCohort - leader302: Failed to transfer leadership in 333.3 ms 09:49:42,925 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader302: LeadershipTransfer request from follower304 failed 09:49:42,926 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Leader - leader302 (Leader): Leadership transfer expired [WARN] [06/30/2025 09:49:42.926] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower304] received dead letter from TestActor[pekko://test/user/leader302]: AppendEntries{term=2, leaderId=leader302, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:49:42,927 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower303: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 09:49:42,927 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower305: Lagging 0 entries behind leader leader302 and reached -1 (of expected -1) 09:49:42,945 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 09:49:42,945 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower303-notifier306#-1066675970] 09:49:42,957 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower303-collector307#875240793] 09:49:42,957 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower303] 09:49:42,957 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower303: terminating thread VirtualThread[#721,follower303-writer-140]/waiting 09:49:42,957 AM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: received 1 commands after 837.0 ms 09:49:42,958 AM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: completed 1 commands in 29.25 μs 09:49:42,958 AM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: journal writer stopped 09:49:42,958 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower303: journal closed 09:49:42,958 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower303: journal closed 09:49:42,958 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: stopped executor with 0 remaining tasks 09:49:42,958 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower303: stopped executor with 0 remaining tasks 09:49:42,958 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower304-notifier308#-1636066212] 09:49:42,958 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower304-collector309#-502378183] 09:49:42,959 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower304] 09:49:42,959 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower305-notifier310#1142375035] 09:49:42,959 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower305-collector311#-475868548] 09:49:42,959 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower305] 09:49:42,959 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - follower305: terminating thread VirtualThread[#723,follower305-writer-142]/waiting 09:49:42,960 AM [follower305-writer-142] [DEBUG] JournalWriteTask - follower305: received 1 commands after 831.9 ms 09:49:42,960 AM [follower305-writer-142] [DEBUG] JournalWriteTask - follower305: completed 1 commands in 20.22 μs 09:49:42,960 AM [follower305-writer-142] [DEBUG] JournalWriteTask - follower305: journal writer stopped 09:49:42,960 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower305: journal closed 09:49:42,960 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower305: journal closed 09:49:42,960 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: stopped executor with 0 remaining tasks 09:49:42,960 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower305: stopped executor with 0 remaining tasks 09:49:42,960 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader302-notifier312#-1306935572] 09:49:42,960 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader302-collector313#-2066082790] 09:49:42,961 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader302] 09:49:42,961 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRaftStorage - leader302: terminating thread VirtualThread[#724,leader302-writer-143]/waiting 09:49:42,961 AM [leader302-writer-143] [DEBUG] JournalWriteTask - leader302: received 1 commands after 830.3 ms 09:49:42,961 AM [leader302-writer-143] [DEBUG] JournalWriteTask - leader302: completed 1 commands in 12.52 μs 09:49:42,961 AM [leader302-writer-143] [DEBUG] JournalWriteTask - leader302: journal writer stopped 09:49:42,961 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader302: journal closed 09:49:42,961 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - leader302: journal closed 09:49:42,961 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader302: stopped executor with 0 remaining tasks 09:49:42,961 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader302: stopped executor with 0 remaining tasks 09:49:42,961 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$b#63188880] 09:49:42,964 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 09:49:42,964 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/30/2025 09:49:42.966] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower315]: org.opendaylight.controller.cluster.common.actor.Monitor@7690caee 09:49:42,967 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower315: Starting recovery with journal batch size 1000 09:49:42,967 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: started executor 09:49:42,967 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: started executor 09:49:42,967 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower315: metadata journal open: replayFrom=1 applyTo=0 09:49:42,968 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower315: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:42,968 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - follower315: journal open: applyTo=0 09:49:42,968 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: journal writer started 09:49:42,968 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: waiting to receive commands 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower315/TermInfo.properties does not exist 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower315/journal-v1-1.log 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower315/metadata-v1-1.log 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: no eligible files found 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower315: SnapshotOffer called. 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower315: initializing from snapshot taken at 2025-06-30T09:49:42.964Z 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=1, votedFor=null 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - follower315: Updated server config: isVoting: true, peers: [PeerInfo [id=follower317, address=pekko://test/user/follower317, votingState=NON_VOTING], PeerInfo [id=leader314, address=pekko://test/user/leader314, votingState=VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=VOTING]] 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - Recovery snapshot applied for follower315 in 57.58 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower315: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:42.964Z 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower315: support() called 09:49:42,969 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower315/snapshot-0000000068625DB6-39757900.tmp [WARN] [06/30/2025 09:49:42.970] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower316]: org.opendaylight.controller.cluster.common.actor.Monitor@58fee7b8 09:49:42,971 AM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - follower316: Starting recovery with journal batch size 1000 09:49:42,971 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: started executor 09:49:42,971 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower316: started executor 09:49:42,971 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower316: metadata journal open: replayFrom=1 applyTo=0 09:49:42,972 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower315/snapshot-0000000068625DB6-39757900.v1 09:49:42,972 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower316: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:42,972 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower316: journal open: applyTo=0 09:49:42,972 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower315/snapshot-0000000068625DB6-39757900.v1 09:49:42,972 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: journal writer started 09:49:42,972 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower315/journal-v1-1.log 09:49:42,972 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: waiting to receive commands 09:49:42,972 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower315/metadata-v1-1.log 09:49:42,972 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower315: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:42.964Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower315/snapshot-0000000068625DB6-39757900.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:42,972 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower315: local snapshot saved in 3.418 ms, deleting Pekko-persisted snapshots 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower316/TermInfo.properties does not exist 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower316/journal-v1-1.log 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower316/metadata-v1-1.log 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: no eligible files found 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower316: SnapshotOffer called. 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - follower316: initializing from snapshot taken at 2025-06-30T09:49:42.964Z 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=1, votedFor=null 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - follower316: Updated server config: isVoting: true, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=follower317, address=pekko://test/user/follower317, votingState=NON_VOTING], PeerInfo [id=leader314, address=pekko://test/user/leader314, votingState=VOTING]] 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - Recovery snapshot applied for follower316 in 54.12 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower316: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:42.964Z 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower316: support() called 09:49:42,973 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower315: Recovery completed in 4.137 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:42,973 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower316/snapshot-0000000068625DB6-39757900.tmp [WARN] [06/30/2025 09:49:42.974] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower317]: org.opendaylight.controller.cluster.common.actor.Monitor@55503e1c 09:49:42,975 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower317: Starting recovery with journal batch size 1000 09:49:42,976 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: started executor 09:49:42,976 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: started executor 09:49:42,976 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=1, votedFor=null 09:49:42,976 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - follower315: Local TermInfo store seeded with TermInfo{term=1} 09:49:42,976 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower315: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6b09a590 09:49:42,976 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower315: Moving last applied index from -1 to -1 09:49:42,976 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] JournalRecovery - follower315: journal recovery completedin 201.7 μs with journalIndex=1 09:49:42,976 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower317: metadata journal open: replayFrom=1 applyTo=0 09:49:42,977 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower315: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:42,977 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower316/snapshot-0000000068625DB6-39757900.v1 09:49:42,977 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower316/snapshot-0000000068625DB6-39757900.v1 09:49:42,977 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower316/journal-v1-1.log 09:49:42,977 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower316/metadata-v1-1.log 09:49:42,977 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower316: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:42.964Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower316/snapshot-0000000068625DB6-39757900.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:42,977 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower316: local snapshot saved in 4.262 ms, deleting Pekko-persisted snapshots 09:49:42,977 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower317: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:42,977 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower317: journal open: applyTo=0 09:49:42,977 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: journal writer started 09:49:42,977 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: waiting to receive commands 09:49:42,978 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower316: Recovery completed in 5.120 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:42,978 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower317/TermInfo.properties does not exist 09:49:42,978 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower317/journal-v1-1.log 09:49:42,978 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower317/metadata-v1-1.log 09:49:42,978 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: no eligible files found 09:49:42,978 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower317: SnapshotOffer called. 09:49:42,978 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower317: initializing from snapshot taken at 2025-06-30T09:49:42.964Z 09:49:42,979 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=1, votedFor=null 09:49:42,979 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower317: Updated server config: isVoting: false, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=leader314, address=pekko://test/user/leader314, votingState=VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=VOTING]] 09:49:42,979 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - Recovery snapshot applied for follower317 in 95.84 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:42,979 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower317: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:42.964Z 09:49:42,979 AM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower317: support() called [WARN] [06/30/2025 09:49:42.979] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader314]: org.opendaylight.controller.cluster.common.actor.Monitor@7185780f 09:49:42,979 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower317/snapshot-0000000068625DB6-39757900.tmp 09:49:42,980 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - leader314: Starting recovery with journal batch size 1000 09:49:42,980 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: started executor 09:49:42,980 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: started executor 09:49:42,980 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader314: metadata journal open: replayFrom=1 applyTo=0 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader314: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRaftStorage - leader314: journal open: applyTo=0 09:49:42,981 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: journal writer started 09:49:42,981 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: waiting to receive commands 09:49:42,981 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=1, votedFor=null 09:49:42,981 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower316: Local TermInfo store seeded with TermInfo{term=1} 09:49:42,981 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower316: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@15829a58 09:49:42,981 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower316: Moving last applied index from -1 to -1 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/leader314/TermInfo.properties does not exist 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/leader314/journal-v1-1.log 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/leader314/metadata-v1-1.log 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: no eligible files found 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - leader314: SnapshotOffer called. 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PekkoRecovery - leader314: initializing from snapshot taken at 2025-06-30T09:49:42.964Z 09:49:42,981 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - follower316: journal recovery completedin 254.1 μs with journalIndex=1 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=1, votedFor=null 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - leader314: Updated server config: isVoting: true, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=follower317, address=pekko://test/user/follower317, votingState=NON_VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=VOTING]] 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - Recovery snapshot applied for leader314 in 59.62 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - leader314: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:42.964Z 09:49:42,981 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader314: support() called 09:49:42,982 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower317/snapshot-0000000068625DB6-39757900.v1 09:49:42,982 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/leader314/snapshot-0000000068625DB6-39757900.tmp 09:49:42,982 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower317/snapshot-0000000068625DB6-39757900.v1 09:49:42,982 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower316: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:42,982 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower317/journal-v1-1.log 09:49:42,982 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower317/metadata-v1-1.log 09:49:42,982 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower317: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:42.964Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/follower317/snapshot-0000000068625DB6-39757900.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:42,982 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower317: local snapshot saved in 2.977 ms, deleting Pekko-persisted snapshots 09:49:42,982 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower317: Recovery completed in 3.909 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:42,984 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/leader314/snapshot-0000000068625DB6-39757900.v1 09:49:42,984 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/leader314/snapshot-0000000068625DB6-39757900.v1 09:49:42,984 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/leader314/journal-v1-1.log 09:49:42,984 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/leader314/metadata-v1-1.log 09:49:42,985 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader314: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:42.964Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6931457978896976672/leader314/snapshot-0000000068625DB6-39757900.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:42,985 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - leader314: local snapshot saved in 3.129 ms, deleting Pekko-persisted snapshots 09:49:42,985 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader314: Recovery completed in 3.939 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:42,985 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=1, votedFor=null 09:49:42,985 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower317: Local TermInfo store seeded with TermInfo{term=1} 09:49:42,985 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower317: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@429fd59 09:49:42,985 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower317: Moving last applied index from -1 to -1 09:49:42,985 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - follower317: journal recovery completedin 156.0 μs with journalIndex=1 09:49:42,986 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower317: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:42,987 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=1, votedFor=null 09:49:42,987 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader314: Local TermInfo store seeded with TermInfo{term=1} 09:49:42,987 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader314: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@695079af 09:49:42,987 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader314: Moving last applied index from -1 to -1 09:49:42,987 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - leader314: journal recovery completedin 221.7 μs with journalIndex=1 09:49:42,988 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader314: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:43,335 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:49:43,335 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - leader314 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.12:17355, Removed) 09:49:43,335 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - leader314 (Follower): Received ElectionTimeout - switching to Candidate 09:49:43,335 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader314 (Candidate): Election: Candidate has following voting peers: [follower315, follower316] 09:49:43,339 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=2, votedFor=leader314 09:49:43,339 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader314 (Candidate): Starting new election term 2 09:49:43,339 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader314 (Candidate): Sending RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} to peer follower315 09:49:43,339 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader314 (Candidate): Sending RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} to peer follower316 09:49:43,339 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - leader314 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:49:43,339 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower315 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:43,339 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower316 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:43,339 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower316 (Follower): Cluster state: [] 09:49:43,339 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower316 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:43,339 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower315 (Follower): Cluster state: [] 09:49:43,339 AM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower316 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:43,339 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower315 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:43,339 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower315 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:43,342 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=2, votedFor=null 09:49:43,342 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower316 (Follower): In requestVote: RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:43,343 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=2, votedFor=null 09:49:43,343 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower315 (Follower): In requestVote: RequestVote{term=2, candidateId=leader314, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:43,346 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=2, votedFor=leader314 09:49:43,347 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower316 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:43,347 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader314 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:49:43,347 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader314 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:49:43,347 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Election: Leader has following peers: [follower315, follower317, follower316] 09:49:43,347 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:43,347 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:43,347 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:43,347 AM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower317 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 09:49:43,347 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader314 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:49:43,347 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower316: Last sync leader does not match current leader leader314, need to catch up to -1 09:49:43,347 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=2, votedFor=leader314 09:49:43,347 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower315 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} [WARN] [06/30/2025 09:49:43.347] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=false] 09:49:43,348 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower315: Last sync leader does not match current leader leader314, need to catch up to -1 [WARN] [06/30/2025 09:49:43.348] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower315] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower315, initialSyncDone=false] 09:49:43,348 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower316 updated: matchIndex: -1, nextIndex: 0 09:49:43,348 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:43,348 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower315 updated: matchIndex: -1, nextIndex: 0 09:49:43,348 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:43,349 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=2, votedFor=null 09:49:43,349 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower317: Last sync leader does not match current leader leader314, need to catch up to -1 09:49:43,349 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower317 updated: matchIndex: -1, nextIndex: 0 [WARN] [06/30/2025 09:49:43.349] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=false] 09:49:43,349 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:43,390 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:49:43,390 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 09:49:43,390 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader314: Persist data index=0 term=2 command=zero 09:49:43,390 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader314: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:49:43,390 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 09:49:43,390 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,390 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower315 09:49:43,390 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower315: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:49:43,390 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: received 1 commands after 409.6 ms 09:49:43,390 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,390 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower317 09:49:43,390 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower317: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:49:43,391 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,391 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:49:43,391 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 09:49:43,391 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): Number of entries to be appended = 1 09:49:43,391 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:49:43,391 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 09:49:43,391 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): Append entry to log zero 09:49:43,391 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower315: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:49:43,391 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower317 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:49:43,391 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower317 (Follower): Number of entries to be appended = 1 09:49:43,391 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: received 1 commands after 422.6 ms 09:49:43,391 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower317 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 09:49:43,391 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower317 (Follower): Append entry to log zero 09:49:43,391 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower317: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:49:43,391 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower317 (Follower): Log size is now 1 09:49:43,391 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: received 1 commands after 413.6 ms 09:49:43,391 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower317 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower317, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:49:43,391 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached -1 (of expected -1) 09:49:43,391 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryStoreCompleter - follower317: awaiting more completions to resolve 1 deferred callback(s) 09:49:43,391 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower317 updated: matchIndex: 0, nextIndex: 1 09:49:43,391 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,391 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): Log size is now 1 09:49:43,391 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower315 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower315, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:49:43,391 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached -1 (of expected -1) 09:49:43,391 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryStoreCompleter - follower315: awaiting more completions to resolve 1 deferred callback(s) 09:49:43,391 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower315 updated: matchIndex: 0, nextIndex: 1 09:49:43,392 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,394 AM [leader314-writer-147] [DEBUG] EntryStoreCompleter - leader314: 1 completion(s) pending 09:49:43,394 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: completed 1 commands in 3.642 ms 09:49:43,394 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryStoreCompleter - leader314: running 1 completion(s) 09:49:43,394 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: waiting to receive commands 09:49:43,394 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader314: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:49:43,394 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader314 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 09:49:43,394 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader314 (Leader): Setting last applied to 0 09:49:43,394 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader314: Moving last applied index from -1 to 0 09:49:43,394 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader314: Applying state for log index 0 data zero 09:49:43,394 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - leader314: applyState called: zero 09:49:43,394 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: received 1 commands after 346.3 μs 09:49:43,395 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: completed 1 commands in 183.3 μs 09:49:43,395 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: waiting to receive commands 09:49:43,395 AM [follower317-writer-146] [DEBUG] EntryStoreCompleter - follower317: 1 completion(s) pending 09:49:43,395 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: completed 1 commands in 3.758 ms 09:49:43,395 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryStoreCompleter - follower317: running 1 completion(s) 09:49:43,395 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: waiting to receive commands 09:49:43,395 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower317: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:49:43,395 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryStoreCompleter - follower317: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0882d58@407bdb0b 09:49:43,395 AM [follower315-writer-144] [DEBUG] EntryStoreCompleter - follower315: 1 completion(s) pending 09:49:43,395 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: completed 1 commands in 4.125 ms 09:49:43,395 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: waiting to receive commands 09:49:43,395 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryStoreCompleter - follower315: running 1 completion(s) 09:49:43,395 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower315: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:49:43,395 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryStoreCompleter - follower315: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0882d58@4ab52c9 [WARN] [06/30/2025 09:49:43.395] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=true] [WARN] [06/30/2025 09:49:43.395] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower315] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower315, initialSyncDone=true] 09:49:43,465 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 09:49:43,466 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 09:49:43,586 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 09:49:43,586 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower315 (Follower): Commit index set to 0 09:49:43,586 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader314 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=2, leaderId=leader314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 09:49:43,586 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:49:43,586 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower315 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 09:49:43,586 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower315 (Follower): Setting last applied to 0 09:49:43,586 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower315: Moving last applied index from -1 to 0 09:49:43,586 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower315: Applying state for log index 0 data zero 09:49:43,586 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower315: applyState called: zero 09:49:43,586 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower317 (Follower): Commit index set to 0 09:49:43,586 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:49:43,586 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower317 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 09:49:43,587 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower317 (Follower): Setting last applied to 0 09:49:43,587 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower317: Moving last applied index from -1 to 0 09:49:43,587 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower317: Applying state for log index 0 data zero 09:49:43,587 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower317: applyState called: zero 09:49:43,587 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: received 1 commands after 191.6 ms 09:49:43,587 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: completed 1 commands in 176.5 μs 09:49:43,587 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: waiting to receive commands 09:49:43,587 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: received 1 commands after 192.0 ms 09:49:43,587 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: completed 1 commands in 200.6 μs 09:49:43,587 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: waiting to receive commands 09:49:43,591 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 09:49:43,591 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader314: onShutDown 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader314: Initiating leader transfer 09:49:43,592 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower317: onLeaderTransitioning: LeaderTransitioning [leaderId=leader314] 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader314: pauseLeader successfully completed - doing transfer 09:49:43,592 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower315: onLeaderTransitioning: LeaderTransitioning [leaderId=leader314] 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader314 (Leader): Attempting to transfer leadership 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 09:49:43,592 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:49:43,592 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader314 (Leader): tryToCompleteLeadershipTransfer: followerId: follower317, matchIndex: 0, lastIndex: 0, isVoting: false 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader314 (Leader): tryToCompleteLeadershipTransfer: followerId: follower315, matchIndex: 0, lastIndex: 0, isVoting: true 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader314 (Leader): Follower's log matches - sending ElectionTimeout 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,592 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:49:43,592 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader314 (Leader): sendAppendEntries: 0 is present for follower follower316 09:49:43,593 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader314 (Leader): Leader transfer complete 09:49:43,593 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader314: leader transfer complete - waiting for new leader 09:49:43,593 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader leader314 and reached 0 (of expected -1) 09:49:43,593 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:49:43,593 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower315 (Follower): Received TimeoutNow - switching to Candidate 09:49:43,593 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - follower315 (Candidate): Election: Candidate has following voting peers: [leader314, follower316] 09:49:43,597 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=3, votedFor=follower315 09:49:43,597 AM [test-pekko.actor.default-dispatcher-11] [INFO] Candidate - follower315 (Candidate): Starting new election term 3 09:49:43,597 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - follower315 (Candidate): Sending RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} to peer leader314 09:49:43,597 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - follower315 (Candidate): Sending RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} to peer follower316 09:49:43,597 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader314 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:43,597 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - follower315 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 09:49:43,597 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader314 (Leader): Cluster state: [] 09:49:43,597 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader314 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 09:49:43,597 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower316 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:43,597 AM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractLeader - leader314 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2}" message is greater than leader's term 2 - switching to Follower 09:49:43,597 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower316 (Follower): Cluster state: [] 09:49:43,597 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower316 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 09:49:43,597 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower316 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2}" message is greater than follower's term 2 - updating term 09:49:43,600 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=3, votedFor=null 09:49:43,600 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader314 (Leader): Leadership transfer in progress - processing RequestVote 09:49:43,600 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader314 (Leader): In requestVote: RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 09:49:43,601 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=3, votedFor=null 09:49:43,601 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower316 (Follower): In requestVote: RequestVote{term=3, candidateId=follower315, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:43,603 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader314: Set currentTerm=3, votedFor=follower315 09:49:43,603 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader314 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 09:49:43,603 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - follower315 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 09:49:43,603 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - follower315 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 09:49:43,603 AM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader314 (Leader) :- Switching from behavior Leader to Follower, election term: 3 09:49:43,603 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): Election: Leader has following peers: [follower317, leader314, follower316] 09:49:43,603 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower317: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,603 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower leader314: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,603 AM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower317 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower315, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 09:49:43,604 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - leader314: Last sync leader does not match current leader follower315, need to catch up to -1 09:49:43,604 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader314: leader changed to follower315 09:49:43,604 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorLeadershipTransferCohort - leader314: Successfully transferred leadership to follower315 in 12.03 ms 09:49:43,604 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader314: leader transfer succeeded - sending PoisonPill 09:49:43,604 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=3, votedFor=follower315 09:49:43,604 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower316 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} [WARN] [06/30/2025 09:49:43.604] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/leader314] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader314, initialSyncDone=false] 09:49:43,604 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRaftStorage - leader314: terminating thread VirtualThread[#728,leader314-writer-147]/waiting 09:49:43,604 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower316: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,604 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: received 1 commands after 209.5 ms 09:49:43,604 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - follower315 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 09:49:43,604 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: completed 1 commands in 24.37 μs 09:49:43,604 AM [leader314-writer-147] [DEBUG] JournalWriteTask - leader314: journal writer stopped 09:49:43,604 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader314 updated: matchIndex: 0, nextIndex: 1 09:49:43,604 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader314: journal closed 09:49:43,604 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower leader314: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:43,604 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - leader314: journal closed 09:49:43,604 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader314: stopped executor with 0 remaining tasks 09:49:43,604 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader314: stopped executor with 0 remaining tasks [WARN] [06/30/2025 09:49:43.605] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/leader314] received dead letter from TestActor[pekko://test/user/follower315]: AppendEntries{term=3, leaderId=follower315, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:49:43,606 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower317: Set currentTerm=3, votedFor=null 09:49:43,606 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower317: Last sync leader does not match current leader follower315, need to catch up to -1 09:49:43,606 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower317 updated: matchIndex: 0, nextIndex: 1 09:49:43,606 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower317: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:43.606] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=false] 09:49:43,606 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) [WARN] [06/30/2025 09:49:43.606] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower317] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower317, initialSyncDone=true] 09:49:44,115 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower316: Last sync leader does not match current leader follower315, need to catch up to -1 [WARN] [06/30/2025 09:49:44.115] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader314] received dead letter from TestActor[pekko://test/user/follower315]: AppendEntries{term=3, leaderId=follower315, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:49:44,115 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower317: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) [WARN] [06/30/2025 09:49:44.116] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=false] 09:49:44,116 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower315 (Leader): sendAppendEntries: 0 is present for follower follower316 09:49:44,116 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower315 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=3, leaderId=follower315, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower316 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=follower315, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower316 (Follower): Number of entries to be appended = 1 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower316 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower316 (Follower): Append entry to log zero 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower316: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower316 (Follower): Log size is now 1 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower316 (Follower): Commit index set to 0 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower316 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower316, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower316: Lagging 0 entries behind leader follower315 and reached 0 (of expected -1) 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower316 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower316 (Follower): Setting last applied to 0 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower316: Moving last applied index from -1 to 0 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower316: Applying state for log index 0 data zero 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower316: applyState called: zero 09:49:44,116 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: received 1 commands after 1.144 s 09:49:44,116 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower315 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower316 updated: matchIndex: 0, nextIndex: 1 09:49:44,116 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower315 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:44,116 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryStoreCompleter - follower316: awaiting more completions to resolve 1 deferred callback(s) 09:49:44,121 AM [follower316-writer-145] [DEBUG] EntryStoreCompleter - follower316: 1 completion(s) pending 09:49:44,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryStoreCompleter - follower316: running 1 completion(s) 09:49:44,121 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: completed 1 commands in 4.621 ms 09:49:44,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower316: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:49:44,121 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: received 1 commands after 6.340 μs 09:49:44,121 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryStoreCompleter - follower316: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0882d58@611237f3 09:49:44,121 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: completed 1 commands in 109.3 μs 09:49:44,121 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: waiting to receive commands [WARN] [06/30/2025 09:49:44.121] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=true] 09:49:44,145 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower316: onLeaderTransitioning: LeaderTransitioning [leaderId=leader314] 09:49:44,145 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 09:49:44,145 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower316 starting 09:49:44,145 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower316: onShutDown 09:49:44,146 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower316: terminating thread VirtualThread[#726,follower316-writer-145]/waiting 09:49:44,146 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: received 1 commands after 24.72 ms 09:49:44,146 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: completed 1 commands in 22.80 μs 09:49:44,146 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: journal writer stopped 09:49:44,146 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower316: journal closed 09:49:44,146 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower316: journal closed 09:49:44,146 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower316: stopped executor with 0 remaining tasks 09:49:44,146 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower316: stopped executor with 0 remaining tasks 09:49:44,146 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower316 ending 09:49:44,146 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 09:49:44,146 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower315-notifier318#1065901678] 09:49:44,158 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower315-collector319#-2122937523] 09:49:44,158 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower315] 09:49:44,158 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower315: terminating thread VirtualThread[#725,follower315-writer-144]/waiting 09:49:44,158 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: received 1 commands after 571.3 ms 09:49:44,158 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: completed 1 commands in 23.15 μs 09:49:44,158 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: journal writer stopped 09:49:44,158 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower315: journal closed 09:49:44,158 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower315: journal closed 09:49:44,158 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower315: stopped executor with 0 remaining tasks 09:49:44,158 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower315: stopped executor with 0 remaining tasks 09:49:44,159 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower316-notifier320#1612800648] 09:49:44,159 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower316-collector321#738274306] 09:49:44,159 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower316] 09:49:44,159 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower317-notifier322#1795949593] 09:49:44,159 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower317-collector323#-342125156] 09:49:44,160 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower317] 09:49:44,160 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower317: terminating thread VirtualThread[#727,follower317-writer-146]/waiting 09:49:44,160 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: received 1 commands after 572.7 ms 09:49:44,160 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: completed 1 commands in 19.98 μs 09:49:44,160 AM [follower317-writer-146] [DEBUG] JournalWriteTask - follower317: journal writer stopped 09:49:44,160 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower317: journal closed 09:49:44,160 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower317: journal closed 09:49:44,160 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower317: stopped executor with 0 remaining tasks 09:49:44,160 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower317: stopped executor with 0 remaining tasks 09:49:44,160 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader314-notifier324#1680727659] 09:49:44,161 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader314-collector325#787425214] 09:49:44,161 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader314] 09:49:44,163 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [06/30/2025 09:49:44.165] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader326]: org.opendaylight.controller.cluster.common.actor.Monitor@54fa162e 09:49:44,166 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - leader326: Starting recovery with journal batch size 1000 09:49:44,166 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader326: started executor 09:49:44,166 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader326: started executor 09:49:44,166 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader326: metadata journal open: replayFrom=1 applyTo=0 09:49:44,167 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader326: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:44,167 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader326: journal open: applyTo=0 09:49:44,167 AM [leader326-writer-148] [DEBUG] JournalWriteTask - leader326: journal writer started 09:49:44,167 AM [leader326-writer-148] [DEBUG] JournalWriteTask - leader326: waiting to receive commands 09:49:44,168 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2461646536045479618/leader326/TermInfo.properties does not exist 09:49:44,168 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader326: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2461646536045479618/leader326/journal-v1-1.log 09:49:44,168 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader326: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest2461646536045479618/leader326/metadata-v1-1.log 09:49:44,169 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader326: no eligible files found 09:49:44,169 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader326: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:44,171 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader326: Set currentTerm=0, votedFor=null 09:49:44,171 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader326: Local TermInfo store seeded with TermInfo{term=0} 09:49:44,172 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader326: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@70eb6c7 09:49:44,172 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader326: Moving last applied index from -1 to -1 09:49:44,172 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - leader326: journal recovery completedin 239.5 μs with journalIndex=1 09:49:44,172 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:49:44,172 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - leader326 (Follower): Received TimeoutNow - switching to Candidate 09:49:44,172 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader326 (Candidate): Election: Candidate has following voting peers: [] 09:49:44,175 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader326: Set currentTerm=1, votedFor=leader326 09:49:44,175 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - leader326 (Candidate): Starting new election term 1 09:49:44,175 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorBehavior - leader326 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 09:49:44,175 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader326 (Candidate): Received ElectionTimeout 09:49:44,175 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader326 (Leader): Election: Leader has following peers: [] 09:49:44,175 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorBehavior - leader326 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 09:49:44,176 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader326 starting 09:49:44,176 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader326: onShutDown 09:49:44,176 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - leader326: terminating thread VirtualThread[#729,leader326-writer-148]/waiting 09:49:44,176 AM [leader326-writer-148] [DEBUG] JournalWriteTask - leader326: received 1 commands after 9.335 ms 09:49:44,176 AM [leader326-writer-148] [DEBUG] JournalWriteTask - leader326: completed 1 commands in 18.76 μs 09:49:44,176 AM [leader326-writer-148] [DEBUG] JournalWriteTask - leader326: journal writer stopped 09:49:44,176 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader326: journal closed 09:49:44,176 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader326: journal closed 09:49:44,177 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader326: stopped executor with 0 remaining tasks 09:49:44,177 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader326: stopped executor with 0 remaining tasks 09:49:44,177 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader326 ending 09:49:44,177 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 09:49:44,177 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader326-collector330#-223016019] 09:49:44,178 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader326] 09:49:44,179 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 09:49:44,179 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/30/2025 09:49:44.181] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower332]: org.opendaylight.controller.cluster.common.actor.Monitor@18199c23 09:49:44,182 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower332: Starting recovery with journal batch size 1000 09:49:44,182 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower332: started executor 09:49:44,182 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower332: started executor 09:49:44,182 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower332: metadata journal open: replayFrom=1 applyTo=0 09:49:44,182 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower332: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:44,182 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower332: journal open: applyTo=0 09:49:44,183 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: journal writer started 09:49:44,183 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: waiting to receive commands 09:49:44,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower332/TermInfo.properties does not exist 09:49:44,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower332/journal-v1-1.log 09:49:44,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower332/metadata-v1-1.log 09:49:44,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: no eligible files found 09:49:44,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower332: SnapshotOffer called. 09:49:44,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower332: initializing from snapshot taken at 2025-06-30T09:49:44.179Z 09:49:44,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=1, votedFor=null 09:49:44,183 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower332: Updated server config: isVoting: true, peers: [PeerInfo [id=leader331, address=pekko://test/user/leader331, votingState=VOTING], PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=VOTING], PeerInfo [id=follower334, address=pekko://test/user/follower334, votingState=NON_VOTING]] 09:49:44,184 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower332 in 67.48 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:44,184 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower332: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:44.179Z 09:49:44,184 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower332: support() called 09:49:44,184 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower332/snapshot-0000000068625DB8-0AAB52C0.tmp [WARN] [06/30/2025 09:49:44.184] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower333]: org.opendaylight.controller.cluster.common.actor.Monitor@52886c83 09:49:44,185 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower333: Starting recovery with journal batch size 1000 09:49:44,185 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower333: started executor 09:49:44,185 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower333: started executor 09:49:44,185 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower333: metadata journal open: replayFrom=1 applyTo=0 09:49:44,186 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower333: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:44,186 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower333: journal open: applyTo=0 09:49:44,186 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: journal writer started 09:49:44,186 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: waiting to receive commands 09:49:44,186 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower332/snapshot-0000000068625DB8-0AAB52C0.v1 09:49:44,186 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower332/snapshot-0000000068625DB8-0AAB52C0.v1 09:49:44,186 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower332/journal-v1-1.log 09:49:44,186 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower332/metadata-v1-1.log 09:49:44,186 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:44.179Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower332/snapshot-0000000068625DB8-0AAB52C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:44,186 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower332: local snapshot saved in 2.571 ms, deleting Pekko-persisted snapshots 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower333/TermInfo.properties does not exist 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower333/journal-v1-1.log 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower333/metadata-v1-1.log 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: no eligible files found 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower333: SnapshotOffer called. 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower333: initializing from snapshot taken at 2025-06-30T09:49:44.179Z 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=1, votedFor=null 09:49:44,187 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower332: Recovery completed in 3.567 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower333: Updated server config: isVoting: true, peers: [PeerInfo [id=leader331, address=pekko://test/user/leader331, votingState=VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING], PeerInfo [id=follower334, address=pekko://test/user/follower334, votingState=NON_VOTING]] 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower333 in 53.40 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower333: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:44.179Z 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower333: support() called 09:49:44,187 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower333/snapshot-0000000068625DB8-0AAB52C0.tmp [WARN] [06/30/2025 09:49:44.188] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower334]: org.opendaylight.controller.cluster.common.actor.Monitor@6c9a1c31 09:49:44,189 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - follower334: Starting recovery with journal batch size 1000 09:49:44,189 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: started executor 09:49:44,189 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: started executor 09:49:44,189 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower334: metadata journal open: replayFrom=1 applyTo=0 09:49:44,190 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower334: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:44,190 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - follower334: journal open: applyTo=0 09:49:44,190 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: journal writer started 09:49:44,190 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: waiting to receive commands 09:49:44,190 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower334/TermInfo.properties does not exist 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower334/journal-v1-1.log 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower334/metadata-v1-1.log 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: no eligible files found 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower334: SnapshotOffer called. 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower334: initializing from snapshot taken at 2025-06-30T09:49:44.179Z 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower334: Set currentTerm=1, votedFor=null 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - follower334: Updated server config: isVoting: false, peers: [PeerInfo [id=leader331, address=pekko://test/user/leader331, votingState=VOTING], PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING]] 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - Recovery snapshot applied for follower334 in 66.57 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower334: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:44.179Z 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower334: support() called 09:49:44,191 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower334/snapshot-0000000068625DB8-0AAB52C0.tmp 09:49:44,193 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActor - leader331: Starting recovery with journal batch size 1000 09:49:44,193 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: started executor 09:49:44,193 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: started executor 09:49:44,193 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower334/snapshot-0000000068625DB8-0AAB52C0.v1 09:49:44,193 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower334/snapshot-0000000068625DB8-0AAB52C0.v1 09:49:44,193 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower334/journal-v1-1.log 09:49:44,193 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower334/metadata-v1-1.log 09:49:44,193 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower334: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:44.179Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower334/snapshot-0000000068625DB8-0AAB52C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:44,193 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower334: local snapshot saved in 2.324 ms, deleting Pekko-persisted snapshots [WARN] [06/30/2025 09:49:44.193] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader331]: org.opendaylight.controller.cluster.common.actor.Monitor@15da8abe 09:49:44,194 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader331: metadata journal open: replayFrom=1 applyTo=0 09:49:44,194 AM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower334: Recovery completed in 3.073 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:44,194 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=1, votedFor=null 09:49:44,194 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower332: Local TermInfo store seeded with TermInfo{term=1} 09:49:44,194 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower332: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@5a6ae37 09:49:44,194 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower332: Moving last applied index from -1 to -1 09:49:44,194 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader331: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:44,194 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - leader331: journal open: applyTo=0 09:49:44,194 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower333/snapshot-0000000068625DB8-0AAB52C0.v1 09:49:44,194 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: journal writer started 09:49:44,194 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower333/snapshot-0000000068625DB8-0AAB52C0.v1 09:49:44,194 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: waiting to receive commands 09:49:44,194 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower333/journal-v1-1.log 09:49:44,194 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower333/metadata-v1-1.log 09:49:44,194 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:44.179Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/follower333/snapshot-0000000068625DB8-0AAB52C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:44,194 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower332: journal recovery completedin 320.4 μs with journalIndex=1 09:49:44,194 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower333: local snapshot saved in 7.240 ms, deleting Pekko-persisted snapshots 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/leader331/TermInfo.properties does not exist 09:49:44,195 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower333: Recovery completed in 8.075 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/leader331/journal-v1-1.log 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/leader331/metadata-v1-1.log 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: no eligible files found 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - leader331: SnapshotOffer called. 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - leader331: initializing from snapshot taken at 2025-06-30T09:49:44.179Z 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader331: Set currentTerm=1, votedFor=null 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - leader331: Updated server config: isVoting: true, peers: [PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING], PeerInfo [id=follower334, address=pekko://test/user/follower334, votingState=NON_VOTING]] 09:49:44,195 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower332: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for leader331 in 75.59 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader331: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:44.179Z 09:49:44,195 AM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - leader331: support() called 09:49:44,196 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/leader331/snapshot-0000000068625DB8-0AAB52C0.tmp 09:49:44,196 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower334: Set currentTerm=1, votedFor=null 09:49:44,196 AM [test-pekko.actor.default-dispatcher-13] [INFO] PekkoRecovery - follower334: Local TermInfo store seeded with TermInfo{term=1} 09:49:44,196 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower334: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@69693388 09:49:44,196 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower334: Moving last applied index from -1 to -1 09:49:44,196 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - follower334: journal recovery completedin 188.1 μs with journalIndex=1 09:49:44,197 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower334: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:44,197 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=1, votedFor=null 09:49:44,197 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower333: Local TermInfo store seeded with TermInfo{term=1} 09:49:44,197 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower333: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@65253210 09:49:44,197 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower333: Moving last applied index from -1 to -1 09:49:44,198 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - follower333: journal recovery completedin 183.8 μs with journalIndex=1 09:49:44,198 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower333: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:44,198 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/leader331/snapshot-0000000068625DB8-0AAB52C0.v1 09:49:44,198 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/leader331/snapshot-0000000068625DB8-0AAB52C0.v1 09:49:44,198 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/leader331/journal-v1-1.log 09:49:44,198 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/leader331/metadata-v1-1.log 09:49:44,198 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:44.179Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest10380806830237506761/leader331/snapshot-0000000068625DB8-0AAB52C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:44,198 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader331: local snapshot saved in 3.016 ms, deleting Pekko-persisted snapshots 09:49:44,199 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader331: Recovery completed in 3.773 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:44,201 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader331: Set currentTerm=1, votedFor=null 09:49:44,201 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader331: Local TermInfo store seeded with TermInfo{term=1} 09:49:44,201 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader331: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6bc235e5 09:49:44,201 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader331: Moving last applied index from -1 to -1 09:49:44,201 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - leader331: journal recovery completedin 162.7 μs with journalIndex=1 09:49:44,201 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader331: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:44,605 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:49:44,605 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - leader331 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.12:17355, Removed) 09:49:44,605 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - leader331 (Follower): Received ElectionTimeout - switching to Candidate 09:49:44,605 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader331 (Candidate): Election: Candidate has following voting peers: [follower333, follower332] 09:49:44,609 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader331: Set currentTerm=2, votedFor=leader331 09:49:44,609 AM [test-pekko.actor.default-dispatcher-13] [INFO] Candidate - leader331 (Candidate): Starting new election term 2 09:49:44,609 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader331 (Candidate): Sending RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} to peer follower333 09:49:44,609 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - leader331 (Candidate): Sending RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} to peer follower332 09:49:44,609 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower333 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:44,609 AM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - leader331 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:49:44,609 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower333 (Follower): Cluster state: [] 09:49:44,609 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower333 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:44,609 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower333 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:44,609 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower332 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:44,609 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower332 (Follower): Cluster state: [] 09:49:44,609 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower332 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:44,609 AM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower332 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:44,612 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=2, votedFor=null 09:49:44,612 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower333 (Follower): In requestVote: RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:44,613 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=2, votedFor=null 09:49:44,613 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower332 (Follower): In requestVote: RequestVote{term=2, candidateId=leader331, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:44,613 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=2, votedFor=leader331 09:49:44,613 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower333 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:44,614 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader331 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:49:44,614 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - leader331 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:49:44,614 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Election: Leader has following peers: [follower333, follower332, follower334] 09:49:44,614 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:44,614 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:44,614 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:44,614 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower333: Last sync leader does not match current leader leader331, need to catch up to -1 09:49:44,614 AM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - leader331 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:49:44,614 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower334 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term [WARN] [06/30/2025 09:49:44.614] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower333] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower333, initialSyncDone=false] 09:49:44,614 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower333 updated: matchIndex: -1, nextIndex: 0 09:49:44,615 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:44,615 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=2, votedFor=leader331 09:49:44,615 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower332 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:44,615 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower332: Last sync leader does not match current leader leader331, need to catch up to -1 09:49:44,616 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower332 updated: matchIndex: -1, nextIndex: 0 09:49:44,616 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:44.616] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower332] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower332, initialSyncDone=false] 09:49:44,616 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower334: Set currentTerm=2, votedFor=null 09:49:44,616 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower334: Last sync leader does not match current leader leader331, need to catch up to -1 09:49:44,616 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower334 updated: matchIndex: -1, nextIndex: 0 09:49:44,616 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:44.616] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower334] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower334, initialSyncDone=false] 09:49:44,654 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:49:44,654 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 09:49:44,654 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 09:49:44,654 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader331: Persist data index=0 term=2 command=zero 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader331: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:44,655 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: received 1 commands after 460.4 ms 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower333: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower332 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower332: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower334 09:49:44,655 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:49:44,655 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): Number of entries to be appended = 1 09:49:44,655 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 09:49:44,655 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): Append entry to log zero 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower334: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:49:44,655 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower332: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:49:44,655 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): Log size is now 1 09:49:44,655 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower332, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:49:44,655 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached -1 (of expected -1) 09:49:44,655 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryStoreCompleter - follower332: awaiting more completions to resolve 1 deferred callback(s) 09:49:44,655 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: received 1 commands after 472.6 ms 09:49:44,655 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower332 updated: matchIndex: 0, nextIndex: 1 09:49:44,655 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:44,655 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 09:49:44,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): Number of entries to be appended = 1 09:49:44,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 09:49:44,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): Append entry to log zero 09:49:44,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower334: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:49:44,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): Log size is now 1 09:49:44,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower334 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower334, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 09:49:44,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached -1 (of expected -1) 09:49:44,656 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: received 1 commands after 465.8 ms 09:49:44,656 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryStoreCompleter - follower334: awaiting more completions to resolve 1 deferred callback(s) 09:49:44,656 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower334 updated: matchIndex: 0, nextIndex: 1 09:49:44,656 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:44,659 AM [leader331-writer-152] [DEBUG] EntryStoreCompleter - leader331: 1 completion(s) pending 09:49:44,660 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: completed 1 commands in 4.775 ms 09:49:44,660 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: waiting to receive commands 09:49:44,660 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryStoreCompleter - leader331: running 1 completion(s) 09:49:44,660 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] ReplicatedLogImpl - leader331: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} 09:49:44,660 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 09:49:44,660 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - leader331 (Leader): Setting last applied to 0 09:49:44,660 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractReplicatedLog - leader331: Moving last applied index from -1 to 0 09:49:44,660 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader331: Applying state for log index 0 data zero 09:49:44,660 AM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - leader331: applyState called: zero 09:49:44,660 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: received 1 commands after 515.9 μs 09:49:44,660 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: completed 1 commands in 120.7 μs 09:49:44,660 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: waiting to receive commands 09:49:44,660 AM [follower334-writer-151] [DEBUG] EntryStoreCompleter - follower334: 1 completion(s) pending 09:49:44,660 AM [follower332-writer-149] [DEBUG] EntryStoreCompleter - follower332: 1 completion(s) pending 09:49:44,660 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: completed 1 commands in 4.454 ms 09:49:44,660 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: waiting to receive commands 09:49:44,660 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: completed 1 commands in 5.039 ms 09:49:44,660 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: waiting to receive commands 09:49:44,660 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryStoreCompleter - follower332: running 1 completion(s) 09:49:44,660 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - follower332: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:49:44,660 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryStoreCompleter - follower332: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0882d58@ee02c44 [WARN] [06/30/2025 09:49:44.661] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower332] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower332, initialSyncDone=true] 09:49:44,661 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryStoreCompleter - follower334: running 1 completion(s) 09:49:44,661 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - follower334: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 09:49:44,661 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryStoreCompleter - follower334: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c0882d58@1bfc59cb [WARN] [06/30/2025 09:49:44.661] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower334] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower334, initialSyncDone=true] 09:49:44,725 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 09:49:44,725 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower333: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 09:49:44,845 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 09:49:44,845 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader331 (Leader): Sending AppendEntries to follower follower333: AppendEntries{term=2, leaderId=leader331, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 09:49:44,846 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): Commit index set to 0 09:49:44,846 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower334 (Follower): Commit index set to 0 09:49:44,846 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:49:44,846 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - follower332 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 09:49:44,846 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:49:44,846 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower332 (Follower): Setting last applied to 0 09:49:44,846 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower334 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 09:49:44,846 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower332: Moving last applied index from -1 to 0 09:49:44,846 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower334 (Follower): Setting last applied to 0 09:49:44,846 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower332: Applying state for log index 0 data zero 09:49:44,846 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower334: Moving last applied index from -1 to 0 09:49:44,846 AM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - follower332: applyState called: zero 09:49:44,846 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower334: Applying state for log index 0 data zero 09:49:44,846 AM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - follower334: applyState called: zero 09:49:44,846 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: received 1 commands after 185.9 ms 09:49:44,846 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: received 1 commands after 186.0 ms 09:49:44,846 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: completed 1 commands in 123.9 μs 09:49:44,846 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: completed 1 commands in 164.6 μs 09:49:44,846 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: waiting to receive commands 09:49:44,846 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: waiting to receive commands 09:49:44,856 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 09:49:44,856 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 09:49:44,856 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 09:49:44,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader331: onRequestLeadership RequestLeadership [requestedFollowerId=follower333, replyTo=Actor[pekko://test/user/$c#-95650644]] 09:49:44,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader331: Initiating leader transfer 09:49:44,856 AM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower333: onLeaderTransitioning: LeaderTransitioning [leaderId=leader331] 09:49:44,856 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower332: onLeaderTransitioning: LeaderTransitioning [leaderId=leader331] 09:49:44,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader331: pauseLeader successfully completed - doing transfer 09:49:44,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Leader - leader331 (Leader): Attempting to transfer leadership 09:49:44,856 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower334: onLeaderTransitioning: LeaderTransitioning [leaderId=leader331] 09:49:44,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:44,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): sendAppendEntries: 0 is present for follower follower333 09:49:44,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:44,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader331 (Leader): Checking sendAppendEntries for follower follower334: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 09:49:44,856 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:49:44,856 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:49:44,965 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:49:44,965 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:49:45,85 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:49:45,85 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:49:45,205 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader331: leader transfer aborted 09:49:45,205 AM [test-pekko.actor.default-dispatcher-6] [WARN] RaftActorLeadershipTransferCohort - leader331: Failed to transfer leadership in 349.3 ms 09:49:45,205 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader331: LeadershipTransfer request from follower333 failed 09:49:45,205 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Leader - leader331 (Leader): Leadership transfer expired 09:49:45,206 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower332: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:49:45,206 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower334: Lagging 0 entries behind leader leader331 and reached 0 (of expected -1) 09:49:45,208 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 09:49:45,208 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower332-notifier335#-1282776648] 09:49:45,219 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower332-collector336#1279778700] 09:49:45,219 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower332] 09:49:45,220 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower332: terminating thread VirtualThread[#730,follower332-writer-149]/waiting 09:49:45,220 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: received 1 commands after 373.2 ms 09:49:45,220 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: completed 1 commands in 32.80 μs 09:49:45,220 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: journal writer stopped 09:49:45,220 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower332: journal closed 09:49:45,220 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower332: journal closed 09:49:45,220 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: stopped executor with 0 remaining tasks 09:49:45,220 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: stopped executor with 0 remaining tasks 09:49:45,220 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower333-notifier337#-1923363812] 09:49:45,221 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower333-collector338#1036495545] 09:49:45,221 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower333] 09:49:45,221 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower333: terminating thread VirtualThread[#731,follower333-writer-150]/waiting 09:49:45,221 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: received 1 commands after 1.035 s 09:49:45,221 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: completed 1 commands in 19.83 μs 09:49:45,221 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: journal writer stopped 09:49:45,221 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower333: journal closed 09:49:45,221 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower333: journal closed 09:49:45,221 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower333: stopped executor with 0 remaining tasks 09:49:45,221 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower333: stopped executor with 0 remaining tasks 09:49:45,222 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower334-notifier339#-903980855] 09:49:45,222 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower334-collector340#1602043588] 09:49:45,222 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower334] 09:49:45,222 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower334: terminating thread VirtualThread[#732,follower334-writer-151]/waiting 09:49:45,222 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: received 1 commands after 375.9 ms 09:49:45,222 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: completed 1 commands in 23.90 μs 09:49:45,222 AM [follower334-writer-151] [DEBUG] JournalWriteTask - follower334: journal writer stopped 09:49:45,223 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower334: journal closed 09:49:45,223 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower334: journal closed 09:49:45,223 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower334: stopped executor with 0 remaining tasks 09:49:45,223 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower334: stopped executor with 0 remaining tasks 09:49:45,223 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader331-notifier341#360750562] 09:49:45,223 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader331-collector342#-249046409] 09:49:45,223 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader331] 09:49:45,224 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - leader331: terminating thread VirtualThread[#733,leader331-writer-152]/waiting 09:49:45,224 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: received 1 commands after 563.4 ms 09:49:45,224 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: completed 1 commands in 19.38 μs 09:49:45,224 AM [leader331-writer-152] [DEBUG] JournalWriteTask - leader331: journal writer stopped 09:49:45,224 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader331: journal closed 09:49:45,224 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - leader331: journal closed 09:49:45,224 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: stopped executor with 0 remaining tasks 09:49:45,224 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader331: stopped executor with 0 remaining tasks 09:49:45,224 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$c#-95650644] 09:49:45,226 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 09:49:45,226 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/30/2025 09:49:45.227] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower344]: org.opendaylight.controller.cluster.common.actor.Monitor@622615aa 09:49:45,228 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActor - follower344: Starting recovery with journal batch size 1000 09:49:45,228 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: started executor 09:49:45,228 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: started executor 09:49:45,229 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower344: metadata journal open: replayFrom=1 applyTo=0 09:49:45,229 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower344: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:45,229 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower344: journal open: applyTo=0 09:49:45,229 AM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: journal writer started 09:49:45,229 AM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: waiting to receive commands 09:49:45,230 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower344/TermInfo.properties does not exist 09:49:45,230 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower344/journal-v1-1.log 09:49:45,230 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower344/metadata-v1-1.log 09:49:45,230 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: no eligible files found 09:49:45,230 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower344: SnapshotOffer called. 09:49:45,230 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower344: initializing from snapshot taken at 2025-06-30T09:49:45.226Z 09:49:45,230 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=1, votedFor=null 09:49:45,230 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - follower344: Updated server config: isVoting: true, peers: [PeerInfo [id=leader343, address=pekko://test/user/leader343, votingState=VOTING], PeerInfo [id=follower346, address=pekko://test/user/follower346, votingState=NON_VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=VOTING]] 09:49:45,231 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for follower344 in 129.0 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:45,231 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower344: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:45.226Z 09:49:45,231 AM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - follower344: support() called 09:49:45,231 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - follower345: Starting recovery with journal batch size 1000 09:49:45,231 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower345: started executor 09:49:45,231 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower345: started executor 09:49:45,231 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower344/snapshot-0000000068625DB9-0D787C80.tmp 09:49:45,231 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower345: metadata journal open: replayFrom=1 applyTo=0 09:49:45,232 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower345: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:45,232 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower345: journal open: applyTo=0 09:49:45,232 AM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: journal writer started 09:49:45,232 AM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: waiting to receive commands [WARN] [06/30/2025 09:49:45.232] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower345]: org.opendaylight.controller.cluster.common.actor.Monitor@5ccfc236 09:49:45,232 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower345/TermInfo.properties does not exist 09:49:45,232 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower345/journal-v1-1.log 09:49:45,232 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower345/metadata-v1-1.log 09:49:45,233 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower345: no eligible files found 09:49:45,233 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower345: SnapshotOffer called. 09:49:45,233 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower345: initializing from snapshot taken at 2025-06-30T09:49:45.226Z 09:49:45,233 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=1, votedFor=null 09:49:45,233 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PeerInfos - follower345: Updated server config: isVoting: true, peers: [PeerInfo [id=leader343, address=pekko://test/user/leader343, votingState=VOTING], PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower346, address=pekko://test/user/follower346, votingState=NON_VOTING]] 09:49:45,233 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - Recovery snapshot applied for follower345 in 73.12 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:45,233 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower345: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:45.226Z 09:49:45,233 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower345: support() called 09:49:45,233 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower345: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower345/snapshot-0000000068625DB9-0D787C80.tmp [WARN] [06/30/2025 09:49:45.234] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower346]: org.opendaylight.controller.cluster.common.actor.Monitor@7687126f 09:49:45,235 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - follower346: Starting recovery with journal batch size 1000 09:49:45,235 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: started executor 09:49:45,235 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: started executor 09:49:45,235 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower346: metadata journal open: replayFrom=1 applyTo=0 09:49:45,236 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower344/snapshot-0000000068625DB9-0D787C80.v1 09:49:45,236 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower344/snapshot-0000000068625DB9-0D787C80.v1 09:49:45,236 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower344/journal-v1-1.log 09:49:45,236 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower344/metadata-v1-1.log 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower346: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower346: journal open: applyTo=0 09:49:45,236 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower344: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:45.226Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower344/snapshot-0000000068625DB9-0D787C80.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:45,236 AM [follower346-writer-155] [DEBUG] JournalWriteTask - follower346: journal writer started 09:49:45,236 AM [follower346-writer-155] [DEBUG] JournalWriteTask - follower346: waiting to receive commands 09:49:45,236 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower344: local snapshot saved in 5.142 ms, deleting Pekko-persisted snapshots 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower346/TermInfo.properties does not exist 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower346/journal-v1-1.log 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower346/metadata-v1-1.log 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: no eligible files found 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower346: SnapshotOffer called. 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower346: initializing from snapshot taken at 2025-06-30T09:49:45.226Z 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower346: Set currentTerm=1, votedFor=null 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower346: Updated server config: isVoting: false, peers: [PeerInfo [id=leader343, address=pekko://test/user/leader343, votingState=VOTING], PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=VOTING]] 09:49:45,236 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower346 in 48.44 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:45,237 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower346: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:45.226Z 09:49:45,237 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower346: support() called 09:49:45,237 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower346/snapshot-0000000068625DB9-0D787C80.tmp [WARN] [06/30/2025 09:49:45.238] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader343]: org.opendaylight.controller.cluster.common.actor.Monitor@2ecbbbe4 09:49:45,238 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower344: Recovery completed in 7.826 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:45,238 AM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - leader343: Starting recovery with journal batch size 1000 09:49:45,238 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: started executor 09:49:45,238 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower345: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower345/snapshot-0000000068625DB9-0D787C80.v1 09:49:45,238 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: started executor 09:49:45,238 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower345: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower345/snapshot-0000000068625DB9-0D787C80.v1 09:49:45,238 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower345/journal-v1-1.log 09:49:45,238 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower345/metadata-v1-1.log 09:49:45,238 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower345: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:45.226Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower345/snapshot-0000000068625DB9-0D787C80.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:45,239 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower346/snapshot-0000000068625DB9-0D787C80.v1 09:49:45,239 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower345: local snapshot saved in 5.649 ms, deleting Pekko-persisted snapshots 09:49:45,239 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower346/snapshot-0000000068625DB9-0D787C80.v1 09:49:45,239 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower346/journal-v1-1.log 09:49:45,239 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower346/metadata-v1-1.log 09:49:45,239 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:45.226Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/follower346/snapshot-0000000068625DB9-0D787C80.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:45,239 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower346: local snapshot saved in 2.134 ms, deleting Pekko-persisted snapshots 09:49:45,239 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower345: Recovery completed in 6.400 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:45,239 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader343: metadata journal open: replayFrom=1 applyTo=0 09:49:45,239 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower346: Recovery completed in 2.707 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:45,240 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader343: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 09:49:45,240 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - leader343: journal open: applyTo=0 09:49:45,240 AM [leader343-writer-156] [DEBUG] JournalWriteTask - leader343: journal writer started 09:49:45,240 AM [leader343-writer-156] [DEBUG] JournalWriteTask - leader343: waiting to receive commands 09:49:45,240 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/leader343/TermInfo.properties does not exist 09:49:45,240 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/leader343/journal-v1-1.log 09:49:45,240 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/leader343/metadata-v1-1.log 09:49:45,240 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: no eligible files found 09:49:45,240 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - leader343: SnapshotOffer called. 09:49:45,241 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRecovery - leader343: initializing from snapshot taken at 2025-06-30T09:49:45.226Z 09:49:45,241 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=1, votedFor=null 09:49:45,241 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - leader343: Updated server config: isVoting: true, peers: [PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower346, address=pekko://test/user/follower346, votingState=NON_VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=VOTING]] 09:49:45,241 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - Recovery snapshot applied for leader343 in 104.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 09:49:45,241 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - leader343: migrating from Pekko persistent-snapshot taken at 2025-06-30T09:49:45.226Z 09:49:45,241 AM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - leader343: support() called 09:49:45,241 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/leader343/snapshot-0000000068625DB9-0D787C80.tmp 09:49:45,242 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=1, votedFor=null 09:49:45,242 AM [test-pekko.actor.default-dispatcher-11] [INFO] PekkoRecovery - follower344: Local TermInfo store seeded with TermInfo{term=1} 09:49:45,242 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower344: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@14daad20 09:49:45,242 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower344: Moving last applied index from -1 to -1 09:49:45,242 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] JournalRecovery - follower344: journal recovery completedin 205.2 μs with journalIndex=1 09:49:45,242 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower344: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:45,243 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=1, votedFor=null 09:49:45,243 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower345: Local TermInfo store seeded with TermInfo{term=1} 09:49:45,243 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower345: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6a6aeb32 09:49:45,243 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower345: Moving last applied index from -1 to -1 09:49:45,244 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower345: journal recovery completedin 177.1 μs with journalIndex=1 09:49:45,244 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower345: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:45,244 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/leader343/snapshot-0000000068625DB9-0D787C80.v1 09:49:45,244 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower346: Set currentTerm=1, votedFor=null 09:49:45,244 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower346: Local TermInfo store seeded with TermInfo{term=1} 09:49:45,244 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/leader343/snapshot-0000000068625DB9-0D787C80.v1 09:49:45,244 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower346: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@409b597d 09:49:45,244 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower346: Moving last applied index from -1 to -1 09:49:45,244 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/leader343/journal-v1-1.log 09:49:45,244 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/leader343/metadata-v1-1.log 09:49:45,244 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-30T09:49:45.226Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest17182903897984873150/leader343/snapshot-0000000068625DB9-0D787C80.v1, size=275, entryCompress=NONE, stateCompress=NONE} 09:49:45,244 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRecovery - leader343: local snapshot saved in 3.386 ms, deleting Pekko-persisted snapshots 09:49:45,244 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - follower346: journal recovery completedin 151.1 μs with journalIndex=1 09:49:45,244 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower346: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:45,245 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader343: Recovery completed in 4.152 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 09:49:45,246 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=1, votedFor=null 09:49:45,246 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader343: Local TermInfo store seeded with TermInfo{term=1} 09:49:45,246 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader343: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@33ec8afb 09:49:45,246 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader343: Moving last applied index from -1 to -1 09:49:45,247 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - leader343: journal recovery completedin 157.4 μs with journalIndex=1 09:49:45,247 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader343: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 09:49:45,665 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:49:45,665 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - leader343 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.12:17355, Removed) 09:49:45,665 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - leader343 (Follower): Received ElectionTimeout - switching to Candidate 09:49:45,665 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader343 (Candidate): Election: Candidate has following voting peers: [follower344, follower345] 09:49:45,669 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=2, votedFor=leader343 09:49:45,669 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - leader343 (Candidate): Starting new election term 2 09:49:45,669 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader343 (Candidate): Sending RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} to peer follower344 09:49:45,669 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader343 (Candidate): Sending RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} to peer follower345 09:49:45,669 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower345 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:45,670 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower344 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:45,670 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower345 (Follower): Cluster state: [] 09:49:45,670 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower345 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:45,670 AM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower345 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:45,670 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower344 (Follower): Cluster state: [] 09:49:45,670 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower344 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:45,670 AM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - follower344 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 09:49:45,670 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorBehavior - leader343 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 09:49:45,673 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=2, votedFor=null 09:49:45,673 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower345 (Follower): In requestVote: RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:45,674 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=2, votedFor=null 09:49:45,674 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower344 (Follower): In requestVote: RequestVote{term=2, candidateId=leader343, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:45,675 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=2, votedFor=leader343 09:49:45,675 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - follower345 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:45,675 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader343 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 09:49:45,675 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - leader343 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 09:49:45,675 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader343 (Leader): Election: Leader has following peers: [follower344, follower346, follower345] 09:49:45,675 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,675 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,675 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,675 AM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - leader343 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 09:49:45,675 AM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - follower346 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 09:49:45,676 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower345: Last sync leader does not match current leader leader343, need to catch up to -1 09:49:45,676 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader343 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower345 updated: matchIndex: -1, nextIndex: 0 09:49:45,676 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:45.676] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower345] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower345, initialSyncDone=false] 09:49:45,677 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=2, votedFor=leader343 09:49:45,677 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower344 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 09:49:45,677 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower344: Last sync leader does not match current leader leader343, need to catch up to -1 09:49:45,677 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower344 updated: matchIndex: -1, nextIndex: 0 09:49:45,677 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:45.677] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower344] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower344, initialSyncDone=false] 09:49:45,677 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower346: Set currentTerm=2, votedFor=null 09:49:45,677 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower346: Last sync leader does not match current leader leader343, need to catch up to -1 09:49:45,677 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader343 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower346 updated: matchIndex: -1, nextIndex: 0 [WARN] [06/30/2025 09:49:45.677] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower346] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower346, initialSyncDone=false] 09:49:45,678 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,700 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 09:49:45,700 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 09:49:45,701 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 09:49:45,701 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower344] 09:49:45,711 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower344: terminating thread VirtualThread[#734,follower344-writer-153]/waiting 09:49:45,711 AM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: received 1 commands after 482.3 ms 09:49:45,712 AM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: completed 1 commands in 25.46 μs 09:49:45,712 AM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: journal writer stopped 09:49:45,712 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower344: journal closed 09:49:45,712 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower344: journal closed 09:49:45,712 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: stopped executor with 0 remaining tasks 09:49:45,712 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: stopped executor with 0 remaining tasks 09:49:45,712 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower346] 09:49:45,722 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower346: terminating thread VirtualThread[#736,follower346-writer-155]/waiting 09:49:45,723 AM [follower346-writer-155] [DEBUG] JournalWriteTask - follower346: received 1 commands after 486.7 ms 09:49:45,723 AM [follower346-writer-155] [DEBUG] JournalWriteTask - follower346: completed 1 commands in 12.86 μs 09:49:45,723 AM [follower346-writer-155] [DEBUG] JournalWriteTask - follower346: journal writer stopped 09:49:45,723 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower346: journal closed 09:49:45,723 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower346: journal closed 09:49:45,723 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: stopped executor with 0 remaining tasks 09:49:45,723 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower346: stopped executor with 0 remaining tasks 09:49:45,723 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 09:49:45,723 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 09:49:45,723 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader343: onRequestLeadership RequestLeadership [requestedFollowerId=follower345, replyTo=Actor[pekko://test/user/$d#-843263389]] 09:49:45,723 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader343: Initiating leader transfer [WARN] [06/30/2025 09:49:45.723] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/leader343]: LeaderTransitioning [leaderId=leader343] [WARN] [06/30/2025 09:49:45.724] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/leader343]: LeaderTransitioning [leaderId=leader343] 09:49:45,724 AM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower345: onLeaderTransitioning: LeaderTransitioning [leaderId=leader343] 09:49:45,724 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader343: pauseLeader successfully completed - doing transfer 09:49:45,724 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader343 (Leader): Attempting to transfer leadership 09:49:45,724 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,724 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:45.724] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [06/30/2025 09:49:45.724] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:49:45,724 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,724 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower345: Lagging 0 entries behind leader leader343 and reached -1 (of expected -1) 09:49:45,724 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader343 (Leader): tryToCompleteLeadershipTransfer: followerId: follower345, matchIndex: -1, lastIndex: -1, isVoting: true 09:49:45,724 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader343 (Leader): Follower's log matches - sending ElectionTimeout [WARN] [06/30/2025 09:49:45.724] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower345] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower345, initialSyncDone=true] 09:49:45,724 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,725 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower346: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,725 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader343 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:45.725] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:49:45,725 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader343 (Leader): Leader transfer complete 09:49:45,725 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader343: leader transfer complete - waiting for new leader [WARN] [06/30/2025 09:49:45.725] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/leader343]: AppendEntries{term=2, leaderId=leader343, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 09:49:45,725 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower345: Lagging 0 entries behind leader leader343 and reached -1 (of expected -1) 09:49:45,725 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 09:49:45,725 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower345 (Follower): Received TimeoutNow - switching to Candidate 09:49:45,725 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - follower345 (Candidate): Election: Candidate has following voting peers: [leader343, follower344] 09:49:45,728 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=3, votedFor=follower345 09:49:45,728 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - follower345 (Candidate): Starting new election term 3 09:49:45,728 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - follower345 (Candidate): Sending RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} to peer leader343 09:49:45,728 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - follower345 (Candidate): Sending RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} to peer follower344 09:49:45,728 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - leader343 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 09:49:45,728 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorBehavior - follower345 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 09:49:45,728 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - leader343 (Leader): Cluster state: [] 09:49:45,728 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - leader343 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 09:49:45,728 AM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractLeader - leader343 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower [WARN] [06/30/2025 09:49:45.728] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/follower345]: RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} 09:49:45,731 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=3, votedFor=null 09:49:45,731 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader343 (Leader): Leadership transfer in progress - processing RequestVote 09:49:45,731 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - leader343 (Leader): In requestVote: RequestVote{term=3, candidateId=follower345, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 09:49:45,733 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader343: Set currentTerm=3, votedFor=follower345 09:49:45,734 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - leader343 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 09:49:45,734 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower345 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 09:49:45,734 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower345 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 09:49:45,734 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader343 (Leader) :- Switching from behavior Leader to Follower, election term: 3 09:49:45,734 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower345 (Leader): Election: Leader has following peers: [leader343, follower344, follower346] 09:49:45,734 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower leader343: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,734 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower follower344: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,734 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - leader343: Last sync leader does not match current leader follower345, need to catch up to -1 09:49:45,734 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader343: leader changed to follower345 09:49:45,734 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorLeadershipTransferCohort - leader343: Successfully transferred leadership to follower345 in 10.94 ms 09:49:45,734 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader343: Leadership transferred successfully to follower345 09:49:45,734 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower follower346: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/30/2025 09:49:45.734] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower344] received dead letter from TestActor[pekko://test/user/follower345]: AppendEntries{term=3, leaderId=follower345, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} [WARN] [06/30/2025 09:49:45.734] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader343] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader343, initialSyncDone=false] 09:49:45,734 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - follower345 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [06/30/2025 09:49:45.734] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower346] received dead letter from TestActor[pekko://test/user/follower345]: AppendEntries{term=3, leaderId=follower345, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 09:49:45,734 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower345 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader343 updated: matchIndex: -1, nextIndex: 0 09:49:45,735 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower345 (Leader): Checking sendAppendEntries for follower leader343: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 09:49:45,774 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 09:49:45,774 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower344-notifier347#-1861115965] 09:49:45,785 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower344-collector348#-702274006] 09:49:45,786 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower345-notifier349#-2002002777] 09:49:45,786 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower345-collector350#-1956691545] 09:49:45,786 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower345] 09:49:45,786 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - follower345: terminating thread VirtualThread[#735,follower345-writer-154]/waiting 09:49:45,786 AM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: received 1 commands after 554.5 ms 09:49:45,787 AM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: completed 1 commands in 32.83 μs 09:49:45,787 AM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: journal writer stopped 09:49:45,787 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower345: journal closed 09:49:45,787 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - follower345: journal closed 09:49:45,787 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: stopped executor with 0 remaining tasks 09:49:45,787 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower345: stopped executor with 0 remaining tasks 09:49:45,787 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower346-notifier351#925342665] 09:49:45,787 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower346-collector352#294810657] 09:49:45,787 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader343-notifier353#176534690] 09:49:45,788 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader343-collector354#613925883] 09:49:45,788 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader343] 09:49:45,788 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] PekkoRaftStorage - leader343: terminating thread VirtualThread[#737,leader343-writer-156]/waiting 09:49:45,788 AM [leader343-writer-156] [DEBUG] JournalWriteTask - leader343: received 1 commands after 548.1 ms 09:49:45,788 AM [leader343-writer-156] [DEBUG] JournalWriteTask - leader343: completed 1 commands in 20.55 μs 09:49:45,788 AM [leader343-writer-156] [DEBUG] JournalWriteTask - leader343: journal writer stopped 09:49:45,788 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader343: journal closed 09:49:45,788 AM [test-pekko.actor.default-dispatcher-16] [INFO] PekkoRaftStorage - leader343: journal closed 09:49:45,788 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: stopped executor with 0 remaining tasks 09:49:45,788 AM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader343: stopped executor with 0 remaining tasks 09:49:45,788 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$d#-843263389] [INFO] [06/30/2025 09:49:45.791] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 09:49:45,792 AM [test-pekko.actor.default-dispatcher-18] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [06/30/2025 09:49:45.792] [test-pekko.actor.default-dispatcher-16] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.170.12:17355) [INFO] [06/30/2025 09:49:45.793] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.12:17355] - Exiting completed [INFO] [06/30/2025 09:49:45.793] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.12:17355] - Shutting down... [INFO] [06/30/2025 09:49:45.793] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.12:17355] - Successfully shut down 09:49:45,794 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.12:17355] - Exiting completed 09:49:45,794 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.12:17355] - Shutting down... 09:49:45,794 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.12:17355] - Successfully shut down [INFO] [06/30/2025 09:49:45.795] [test-pekko.remote.default-remote-dispatcher-7] [pekko://test@10.30.170.12:17355/system/remoting-terminator] Shutting down remote daemon. [INFO] [06/30/2025 09:49:45.795] [test-pekko.remote.default-remote-dispatcher-7] [pekko://test@10.30.170.12:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 09:49:45,795 AM [test-pekko.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. 09:49:45,795 AM [test-pekko.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [06/30/2025 09:49:45.797] [test-pekko.remote.default-remote-dispatcher-7] [pekko://test@10.30.170.12:17355/system/remoting-terminator] Remoting shut down. 09:49:45,797 AM [test-pekko.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [DEBUG] [06/30/2025 09:49:45.797] [test-pekko.actor.internal-dispatcher-2] [EventStream] shutting down: StandardOutLogger