[DEBUG] [06/27/2025 11:16:48.520] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 11:16:48,528 AM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [06/27/2025 11:16:48.528] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [06/27/2025 11:16:48.529] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [06/27/2025 11:16:48.545] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.168:17355] with UID [8309614402574132346] [INFO] [06/27/2025 11:16:48.546] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.168:17355] - Starting up, Pekko version [1.0.3] ... 11:16:48,547 AM [test-pekko.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.168:17355] with UID [8309614402574132346] 11:16:48,547 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.168:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [06/27/2025 11:16:48.549] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.168:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [06/27/2025 11:16:48.549] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.168:17355] - Started up successfully 11:16:48,549 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.168:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 11:16:48,549 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.168:17355] - Started up successfully [INFO] [06/27/2025 11:16:48.555] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.168:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [06/27/2025 11:16:48.555] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.168:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 11:16:48,555 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.168:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 11:16:48,555 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.168:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 11:16:48,559 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 11:16:48,559 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:16:48,566 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - follower278: Starting recovery with journal batch size 1000 11:16:48,566 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: started executor 11:16:48,566 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: started executor 11:16:48,566 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower278: metadata journal open: replayFrom=1 applyTo=0 11:16:48,567 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower278: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:48,567 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - follower278: journal open: applyTo=0 11:16:48,567 AM [follower278-writer-131] [DEBUG] JournalWriteTask - follower278: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3a983506 started 11:16:48,567 AM [follower278-writer-131] [DEBUG] JournalWriteTask - follower278: waiting for more work [WARN] [06/27/2025 11:16:48.568] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower278]: org.opendaylight.controller.cluster.common.actor.Monitor@3d17a5a9 [INFO] [pekkoDeadLetter][06/27/2025 11:16:48.567] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower278] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,568 AM [test-pekko.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower278] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [06/27/2025 11:16:48.569] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower279]: org.opendaylight.controller.cluster.common.actor.Monitor@78152c6f [INFO] [pekkoDeadLetter][06/27/2025 11:16:48.570] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower279] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,570 AM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower279] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower278/TermInfo.properties does not exist 11:16:48,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower278/journal-v1-1.log 11:16:48,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower278/metadata-v1-1.log 11:16:48,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: no eligible files found 11:16:48,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower278: SnapshotOffer called. 11:16:48,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower278: initializing from snapshot taken at 2025-06-27T11:16:48.559Z 11:16:48,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower278: Set currentTerm=1, votedFor=null 11:16:48,570 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - follower278: Updated server config: isVoting: true, peers: [PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=NON_VOTING], PeerInfo [id=leader277, address=pekko://test/user/leader277, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 11:16:48,571 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - Recovery snapshot applied for follower278 in 94.32 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:48,571 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower278: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:48.559Z 11:16:48,571 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower278: support() called 11:16:48,571 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower278/snapshot-00000000685E7DA0-2151A9C0.tmp 11:16:48,572 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower279: Starting recovery with journal batch size 1000 11:16:48,573 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower279: started executor 11:16:48,573 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower279: started executor 11:16:48,573 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower279: metadata journal open: replayFrom=1 applyTo=0 11:16:48,573 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower279: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:48,573 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - follower279: journal open: applyTo=0 11:16:48,573 AM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3fda65f started 11:16:48,573 AM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: waiting for more work 11:16:48,574 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower278/snapshot-00000000685E7DA0-2151A9C0.v1 11:16:48,574 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower278/snapshot-00000000685E7DA0-2151A9C0.v1 11:16:48,574 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower278/journal-v1-1.log 11:16:48,574 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower278/metadata-v1-1.log 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower279/TermInfo.properties does not exist 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower279/journal-v1-1.log 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower279/metadata-v1-1.log 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: no eligible files found 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower279: SnapshotOffer called. 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower279: initializing from snapshot taken at 2025-06-27T11:16:48.559Z 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=1, votedFor=null 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PeerInfos - follower279: Updated server config: isVoting: true, peers: [PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=NON_VOTING], PeerInfo [id=leader277, address=pekko://test/user/leader277, votingState=VOTING], PeerInfo [id=follower278, address=pekko://test/user/follower278, votingState=VOTING]] 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - Recovery snapshot applied for follower279 in 169.0 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower279: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:48.559Z 11:16:48,574 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower279: support() called 11:16:48,575 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower279/snapshot-00000000685E7DA0-2151A9C0.tmp 11:16:48,574 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower278: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:48.559Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower278/snapshot-00000000685E7DA0-2151A9C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:48,575 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower278: local snapshot saved in 4.099 ms, deleting Pekko-persisted snapshots 11:16:48,575 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower278: Recovery completed in 4.879 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [06/27/2025 11:16:48.577] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower280]: org.opendaylight.controller.cluster.common.actor.Monitor@50c1028f 11:16:48,577 AM [test-pekko.actor.default-dispatcher-12] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower280] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/27/2025 11:16:48.577] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower280] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,578 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower280: Starting recovery with journal batch size 1000 11:16:48,578 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: started executor 11:16:48,578 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: started executor 11:16:48,578 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower280: metadata journal open: replayFrom=1 applyTo=0 11:16:48,578 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower280: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:48,578 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower280: journal open: applyTo=0 11:16:48,578 AM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@2755f7bd started 11:16:48,579 AM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: waiting for more work 11:16:48,579 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower280/TermInfo.properties does not exist 11:16:48,579 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower280/journal-v1-1.log 11:16:48,579 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower280/metadata-v1-1.log 11:16:48,579 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: no eligible files found 11:16:48,579 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower280: SnapshotOffer called. 11:16:48,579 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower280: initializing from snapshot taken at 2025-06-27T11:16:48.559Z 11:16:48,579 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=1, votedFor=null 11:16:48,579 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower279/snapshot-00000000685E7DA0-2151A9C0.v1 11:16:48,579 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower279/snapshot-00000000685E7DA0-2151A9C0.v1 11:16:48,579 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower280: Updated server config: isVoting: false, peers: [PeerInfo [id=leader277, address=pekko://test/user/leader277, votingState=VOTING], PeerInfo [id=follower278, address=pekko://test/user/follower278, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 11:16:48,579 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower279/journal-v1-1.log 11:16:48,579 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower279/metadata-v1-1.log 11:16:48,579 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower279: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:48.559Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower279/snapshot-00000000685E7DA0-2151A9C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:48,579 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower279: local snapshot saved in 4.934 ms, deleting Pekko-persisted snapshots 11:16:48,579 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - Recovery snapshot applied for follower280 in 63.43 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:48,579 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower280: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:48.559Z 11:16:48,580 AM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower280: support() called 11:16:48,580 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - follower279: Recovery completed in 5.703 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:48,580 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower280/snapshot-00000000685E7DA0-2151A9C0.tmp 11:16:48,580 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower278: Set currentTerm=1, votedFor=null 11:16:48,580 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower278: Local TermInfo store seeded with TermInfo{term=1} 11:16:48,580 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower278: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@9d58712 11:16:48,580 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower278: Moving last applied index from -1 to -1 11:16:48,580 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - leader277: Starting recovery with journal batch size 1000 11:16:48,580 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: started executor 11:16:48,580 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: started executor 11:16:48,581 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower278: journal recovery completedin 222.6 μs with journalIndex=1 11:16:48,581 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader277: metadata journal open: replayFrom=1 applyTo=0 [WARN] [06/27/2025 11:16:48.581] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader277]: org.opendaylight.controller.cluster.common.actor.Monitor@49273648 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader277: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader277: journal open: applyTo=0 11:16:48,582 AM [leader277-writer-134] [DEBUG] JournalWriteTask - leader277: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3eda7203 started 11:16:48,582 AM [leader277-writer-134] [DEBUG] JournalWriteTask - leader277: waiting for more work 11:16:48,582 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=1, votedFor=null 11:16:48,582 AM [test-pekko.actor.default-dispatcher-14] [INFO] PekkoRecovery - follower279: Local TermInfo store seeded with TermInfo{term=1} 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/leader277/TermInfo.properties does not exist 11:16:48,582 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - follower279: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6d3de1ac 11:16:48,582 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower278: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:48,582 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractReplicatedLog - follower279: Moving last applied index from -1 to -1 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/leader277/journal-v1-1.log 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/leader277/metadata-v1-1.log 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: no eligible files found 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - leader277: SnapshotOffer called. 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - leader277: initializing from snapshot taken at 2025-06-27T11:16:48.559Z 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader277: Set currentTerm=1, votedFor=null 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PeerInfos - leader277: Updated server config: isVoting: true, peers: [PeerInfo [id=follower280, address=pekko://test/user/follower280, votingState=NON_VOTING], PeerInfo [id=follower278, address=pekko://test/user/follower278, votingState=VOTING], PeerInfo [id=follower279, address=pekko://test/user/follower279, votingState=VOTING]] 11:16:48,582 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - Recovery snapshot applied for leader277 in 62.43 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:48,583 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader277: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:48.559Z 11:16:48,583 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader277: support() called 11:16:48,583 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] JournalRecovery - follower279: journal recovery completedin 342.0 μs with journalIndex=1 11:16:48,583 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/leader277/snapshot-00000000685E7DA0-2151A9C0.tmp 11:16:48,583 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower280/snapshot-00000000685E7DA0-2151A9C0.v1 11:16:48,583 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower280/snapshot-00000000685E7DA0-2151A9C0.v1 11:16:48,583 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - follower279: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:48,583 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower280/journal-v1-1.log 11:16:48,583 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower280/metadata-v1-1.log 11:16:48,583 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower280: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:48.559Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/follower280/snapshot-00000000685E7DA0-2151A9C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:48,583 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower280: local snapshot saved in 3.987 ms, deleting Pekko-persisted snapshots [INFO] [pekkoDeadLetter][06/27/2025 11:16:48.584] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader277] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,584 AM [test-pekko.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader277] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,584 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower280: Recovery completed in 4.794 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:48,585 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/leader277/snapshot-00000000685E7DA0-2151A9C0.v1 11:16:48,585 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/leader277/snapshot-00000000685E7DA0-2151A9C0.v1 11:16:48,586 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/leader277/journal-v1-1.log 11:16:48,586 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/leader277/metadata-v1-1.log 11:16:48,586 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader277: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:48.559Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest9017229944346171413/leader277/snapshot-00000000685E7DA0-2151A9C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:48,586 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader277: local snapshot saved in 3.049 ms, deleting Pekko-persisted snapshots 11:16:48,586 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader277: Recovery completed in 3.617 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:48,586 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=1, votedFor=null 11:16:48,586 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower280: Local TermInfo store seeded with TermInfo{term=1} 11:16:48,586 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower280: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@2f2353aa 11:16:48,586 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractReplicatedLog - follower280: Moving last applied index from -1 to -1 11:16:48,586 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] JournalRecovery - follower280: journal recovery completedin 206.6 μs with journalIndex=1 11:16:48,587 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower280: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:48,588 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader277: Set currentTerm=1, votedFor=null 11:16:48,588 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader277: Local TermInfo store seeded with TermInfo{term=1} 11:16:48,588 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader277: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@1d59234b 11:16:48,588 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader277: Moving last applied index from -1 to -1 11:16:48,588 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader277: journal recovery completedin 165.8 μs with journalIndex=1 11:16:48,588 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader277: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:48,920 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:48,920 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - leader277 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.168:17355, Removed) 11:16:48,920 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - leader277 (Follower): Received ElectionTimeout - switching to Candidate 11:16:48,922 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader277 (Candidate): Election: Candidate has following voting peers: [follower278, follower279] 11:16:48,925 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader277: Set currentTerm=2, votedFor=leader277 11:16:48,925 AM [test-pekko.actor.default-dispatcher-18] [INFO] Candidate - leader277 (Candidate): Starting new election term 2 11:16:48,925 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader277 (Candidate): Sending RequestVote{term=2, candidateId=leader277, lastLogIndex=-1, lastLogTerm=-1} to peer follower278 11:16:48,925 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader277 (Candidate): Sending RequestVote{term=2, candidateId=leader277, lastLogIndex=-1, lastLogTerm=-1} to peer follower279 11:16:48,925 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader277 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:16:48,925 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower278 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:48,925 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower278 (Follower): Cluster state: [] 11:16:48,925 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower278 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader277, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:48,925 AM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower278 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader277, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:48,925 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower279 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:48,925 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower279 (Follower): Cluster state: [] 11:16:48,925 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower279 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader277, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:48,925 AM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower279 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader277, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:48,928 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower278: Set currentTerm=2, votedFor=null 11:16:48,928 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower278 (Follower): In requestVote: RequestVote{term=2, candidateId=leader277, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:48,929 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=2, votedFor=null 11:16:48,929 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower279 (Follower): In requestVote: RequestVote{term=2, candidateId=leader277, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:48,931 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower278: Set currentTerm=2, votedFor=leader277 11:16:48,931 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower278 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:48,931 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader277 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 11:16:48,931 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - leader277 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 11:16:48,931 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader277 (Leader): Election: Leader has following peers: [follower280, follower278, follower279] 11:16:48,931 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader277 (Leader): Checking sendAppendEntries for follower follower280: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:48,931 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader277 (Leader): Checking sendAppendEntries for follower follower278: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:48,931 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader277 (Leader): Checking sendAppendEntries for follower follower279: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:48,932 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader277 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:16:48,932 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower280 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader277, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 11:16:48,932 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower278: Last sync leader does not match current leader leader277, need to catch up to -1 11:16:48,932 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower279: Set currentTerm=2, votedFor=leader277 11:16:48,932 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower279 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:48,932 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader277 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower278 updated: matchIndex: -1, nextIndex: 0 11:16:48,932 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader277 (Leader): Checking sendAppendEntries for follower follower278: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:48.932] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower278] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower278, initialSyncDone=false] 11:16:48,932 AM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower278] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/27/2025 11:16:48.932] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower278] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower278] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,935 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower279: Last sync leader does not match current leader leader277, need to catch up to -1 [WARN] [06/27/2025 11:16:48.935] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower279] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower279, initialSyncDone=false] 11:16:48,935 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader277 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower279 updated: matchIndex: -1, nextIndex: 0 11:16:48,935 AM [test-pekko.actor.default-dispatcher-18] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/27/2025 11:16:48.935] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower279] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,935 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractLeader - leader277 (Leader): Checking sendAppendEntries for follower follower279: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:48,937 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower280: Set currentTerm=2, votedFor=null 11:16:48,937 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower280: Last sync leader does not match current leader leader277, need to catch up to -1 [WARN] [06/27/2025 11:16:48.937] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower280] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower280, initialSyncDone=false] 11:16:48,937 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader277 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower280 updated: matchIndex: -1, nextIndex: 0 11:16:48,937 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader277 (Leader): Checking sendAppendEntries for follower follower280: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:48,937 AM [test-pekko.actor.default-dispatcher-12] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/27/2025 11:16:48.937] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower280] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,940 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:16:48,940 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader277 starting 11:16:48,942 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader277: onShutDown 11:16:48,942 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader277: Initiating leader transfer 11:16:48,942 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader277: pauseLeader successfully completed - doing transfer 11:16:48,942 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Leader - leader277 (Leader): Attempting to transfer leadership 11:16:48,942 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActor - follower280: onLeaderTransitioning: LeaderTransitioning [leaderId=leader277] 11:16:48,942 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader277 (Leader): Checking sendAppendEntries for follower follower280: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:48,942 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader277 (Leader): Checking sendAppendEntries for follower follower278: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:48,942 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower279: onLeaderTransitioning: LeaderTransitioning [leaderId=leader277] 11:16:48,942 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader277 (Leader): Checking sendAppendEntries for follower follower279: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:48,942 AM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:48,942 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:48,942 AM [test-pekko.actor.default-dispatcher-17] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [06/27/2025 11:16:48.942] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower280] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower280, initialSyncDone=true] [WARN] [06/27/2025 11:16:48.942] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower279] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower279, initialSyncDone=true] 11:16:48,942 AM [test-pekko.actor.default-dispatcher-17] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/27/2025 11:16:48.942] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower280] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower280] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][06/27/2025 11:16:48.942] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower279] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower279] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,942 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower278: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:48,942 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower278: onLeaderTransitioning: LeaderTransitioning [leaderId=leader277] [WARN] [06/27/2025 11:16:48.942] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower278] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower278, initialSyncDone=true] [INFO] [pekkoDeadLetter][06/27/2025 11:16:48.943] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower278] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower278] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:48,943 AM [test-pekko.actor.default-dispatcher-18] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower278] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 11:16:49,51 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:49,51 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:49,51 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower278: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:49,170 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower278: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:49,170 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:49,170 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:49,290 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader277: leader transfer aborted 11:16:49,290 AM [test-pekko.actor.default-dispatcher-12] [WARN] RaftActorLeadershipTransferCohort - leader277: Failed to transfer leadership in 348.4 ms 11:16:49,290 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader277: leader transfer failed - sending PoisonPill 11:16:49,290 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Leader - leader277 (Leader): Leadership transfer expired 11:16:49,291 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower278: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:49,291 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRaftStorage - leader277: terminating thread VirtualThread[#710,leader277-writer-134]/waiting 11:16:49,291 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower279: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:49,291 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower280: Lagging 0 entries behind leader leader277 and reached -1 (of expected -1) 11:16:49,291 AM [leader277-writer-134] [DEBUG] JournalWriteTask - leader277: received 1 entries in 708.8 ms 11:16:49,291 AM [leader277-writer-134] [DEBUG] JournalWriteTask - leader277: persisted 1 entries in 35.95 μs 11:16:49,291 AM [leader277-writer-134] [DEBUG] JournalWriteTask - leader277: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3eda7203 stopped 11:16:49,291 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader277: journal closed 11:16:49,291 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - leader277: journal closed 11:16:49,291 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader277: stopped executor with 0 remaining tasks 11:16:49,291 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader277: stopped executor with 0 remaining tasks [WARN] [06/27/2025 11:16:49.291] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/leader277] received dead letter from TestActor[pekko://test/user/follower278]: AppendEntriesReply{term=2, success=true, followerId=follower278, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} [WARN] [06/27/2025 11:16:49.291] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/leader277] received dead letter from TestActor[pekko://test/user/follower279]: AppendEntriesReply{term=2, success=true, followerId=follower279, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} [WARN] [06/27/2025 11:16:49.292] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/leader277] received dead letter from TestActor[pekko://test/user/follower280]: AppendEntriesReply{term=2, success=true, followerId=follower280, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 11:16:49,292 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader277 ending 11:16:49,295 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 11:16:49,295 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower278-notifier281#1491783349] 11:16:49,306 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower278-collector282#-488264402] 11:16:49,307 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower278] 11:16:49,307 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower278: terminating thread VirtualThread[#702,follower278-writer-131]/waiting 11:16:49,307 AM [follower278-writer-131] [DEBUG] JournalWriteTask - follower278: received 1 entries in 739.9 ms 11:16:49,307 AM [follower278-writer-131] [DEBUG] JournalWriteTask - follower278: persisted 1 entries in 25.77 μs 11:16:49,307 AM [follower278-writer-131] [DEBUG] JournalWriteTask - follower278: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3a983506 stopped 11:16:49,307 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower278: journal closed 11:16:49,307 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower278: journal closed 11:16:49,307 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower278: stopped executor with 0 remaining tasks 11:16:49,307 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower278: stopped executor with 0 remaining tasks 11:16:49,308 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower279-notifier283#-1361913468] 11:16:49,308 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower279-collector284#688821806] 11:16:49,308 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower279] 11:16:49,308 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower279: terminating thread VirtualThread[#706,follower279-writer-132]/waiting 11:16:49,309 AM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: received 1 entries in 735.3 ms 11:16:49,309 AM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: persisted 1 entries in 20.73 μs 11:16:49,309 AM [follower279-writer-132] [DEBUG] JournalWriteTask - follower279: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3fda65f stopped 11:16:49,309 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower279: journal closed 11:16:49,309 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower279: journal closed 11:16:49,309 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower279: stopped executor with 0 remaining tasks 11:16:49,309 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower279: stopped executor with 0 remaining tasks 11:16:49,309 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower280-notifier285#-1555733079] 11:16:49,309 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower280-collector286#1332159785] 11:16:49,309 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower280] 11:16:49,310 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower280: terminating thread VirtualThread[#707,follower280-writer-133]/waiting 11:16:49,310 AM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: received 1 entries in 731.3 ms 11:16:49,310 AM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: persisted 1 entries in 21.43 μs 11:16:49,310 AM [follower280-writer-133] [DEBUG] JournalWriteTask - follower280: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@2755f7bd stopped 11:16:49,310 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower280: journal closed 11:16:49,310 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower280: journal closed 11:16:49,310 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower280: stopped executor with 0 remaining tasks 11:16:49,310 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower280: stopped executor with 0 remaining tasks 11:16:49,310 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader277-notifier287#73531523] 11:16:49,310 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader277-collector288#586452335] 11:16:49,311 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader277] 11:16:49,314 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 11:16:49,314 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/27/2025 11:16:49.316] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower290]: org.opendaylight.controller.cluster.common.actor.Monitor@64f498ef 11:16:49,316 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActor - follower290: Starting recovery with journal batch size 1000 11:16:49,317 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower290: started executor 11:16:49,317 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower290: started executor 11:16:49,317 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower290: metadata journal open: replayFrom=1 applyTo=0 11:16:49,317 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower290: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:49,317 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower290: journal open: applyTo=0 11:16:49,318 AM [follower290-writer-135] [DEBUG] JournalWriteTask - follower290: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@459150db started 11:16:49,318 AM [follower290-writer-135] [DEBUG] JournalWriteTask - follower290: waiting for more work 11:16:49,318 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower290/TermInfo.properties does not exist 11:16:49,318 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower290/journal-v1-1.log 11:16:49,318 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower290/metadata-v1-1.log 11:16:49,318 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower290: no eligible files found 11:16:49,318 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower290: SnapshotOffer called. 11:16:49,318 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower290: initializing from snapshot taken at 2025-06-27T11:16:49.314Z 11:16:49,318 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower290: Set currentTerm=1, votedFor=null 11:16:49,318 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower290: Updated server config: isVoting: true, peers: [PeerInfo [id=leader289, address=pekko://test/user/leader289, votingState=VOTING], PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=NON_VOTING], PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING]] 11:16:49,319 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for follower290 in 71.83 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:49,319 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower290: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:49.314Z 11:16:49,319 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower290: support() called 11:16:49,319 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower290: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower290/snapshot-00000000685E7DA1-12B74280.tmp [WARN] [06/27/2025 11:16:49.319] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower291]: org.opendaylight.controller.cluster.common.actor.Monitor@3665c88c 11:16:49,321 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - follower291: Starting recovery with journal batch size 1000 11:16:49,321 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: started executor 11:16:49,321 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: started executor 11:16:49,321 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower291: metadata journal open: replayFrom=1 applyTo=0 11:16:49,322 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower290: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower290/snapshot-00000000685E7DA1-12B74280.v1 11:16:49,322 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower290/journal-v1-1.log 11:16:49,322 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower290: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower290/metadata-v1-1.log 11:16:49,322 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower290: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower290/snapshot-00000000685E7DA1-12B74280.v1 11:16:49,322 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower290: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:49.314Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower290/snapshot-00000000685E7DA1-12B74280.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:49,322 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower290: local snapshot saved in 3.214 ms, deleting Pekko-persisted snapshots 11:16:49,322 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower291: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:49,322 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower291: journal open: applyTo=0 11:16:49,323 AM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@2bb0d968 started 11:16:49,323 AM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: waiting for more work 11:16:49,323 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower290: Recovery completed in 4.262 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:49,323 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower291/TermInfo.properties does not exist 11:16:49,323 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower291/journal-v1-1.log 11:16:49,323 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower291/metadata-v1-1.log 11:16:49,323 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: no eligible files found 11:16:49,323 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower291: SnapshotOffer called. 11:16:49,323 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower291: initializing from snapshot taken at 2025-06-27T11:16:49.314Z 11:16:49,323 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=1, votedFor=null 11:16:49,323 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower291: Updated server config: isVoting: true, peers: [PeerInfo [id=leader289, address=pekko://test/user/leader289, votingState=VOTING], PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=NON_VOTING], PeerInfo [id=follower290, address=pekko://test/user/follower290, votingState=VOTING]] 11:16:49,324 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower291 in 99.19 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:49,324 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower291: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:49.314Z 11:16:49,324 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower291: support() called 11:16:49,324 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower291/snapshot-00000000685E7DA1-12B74280.tmp [WARN] [06/27/2025 11:16:49.324] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower292]: org.opendaylight.controller.cluster.common.actor.Monitor@174ebaae 11:16:49,325 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - follower292: Starting recovery with journal batch size 1000 11:16:49,325 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: started executor 11:16:49,325 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: started executor 11:16:49,325 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower292: metadata journal open: replayFrom=1 applyTo=0 11:16:49,325 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower292: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:49,325 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRaftStorage - follower292: journal open: applyTo=0 11:16:49,326 AM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@23d02bbc started 11:16:49,326 AM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: waiting for more work 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower292/TermInfo.properties does not exist 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower292/journal-v1-1.log 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower292/metadata-v1-1.log 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: no eligible files found 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower292: SnapshotOffer called. 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PekkoRecovery - follower292: initializing from snapshot taken at 2025-06-27T11:16:49.314Z 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=1, votedFor=null 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower292: Updated server config: isVoting: false, peers: [PeerInfo [id=leader289, address=pekko://test/user/leader289, votingState=VOTING], PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=follower290, address=pekko://test/user/follower290, votingState=VOTING]] 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - Recovery snapshot applied for follower292 in 58.93 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower292: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:49.314Z 11:16:49,326 AM [test-pekko.actor.default-dispatcher-15] [INFO] MockRaftActor - follower292: support() called 11:16:49,327 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower292/snapshot-00000000685E7DA1-12B74280.tmp [WARN] [06/27/2025 11:16:49.327] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader289]: org.opendaylight.controller.cluster.common.actor.Monitor@39cbf1db 11:16:49,327 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower290: Set currentTerm=1, votedFor=null 11:16:49,327 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower290: Local TermInfo store seeded with TermInfo{term=1} 11:16:49,327 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower290: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@61085473 11:16:49,328 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower290: Moving last applied index from -1 to -1 11:16:49,328 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActor - leader289: Starting recovery with journal batch size 1000 11:16:49,328 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader289: started executor 11:16:49,328 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader289: started executor 11:16:49,328 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower290: journal recovery completedin 286.8 μs with journalIndex=1 11:16:49,328 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader289: metadata journal open: replayFrom=1 applyTo=0 11:16:49,328 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower290: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:49,328 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader289: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:49,328 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - leader289: journal open: applyTo=0 11:16:49,329 AM [leader289-writer-138] [DEBUG] JournalWriteTask - leader289: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@4ef71faf started 11:16:49,329 AM [leader289-writer-138] [DEBUG] JournalWriteTask - leader289: waiting for more work 11:16:49,329 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower291/snapshot-00000000685E7DA1-12B74280.v1 11:16:49,329 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower291/journal-v1-1.log 11:16:49,329 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower291/metadata-v1-1.log 11:16:49,329 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower291/snapshot-00000000685E7DA1-12B74280.v1 11:16:49,329 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower291: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:49.314Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower291/snapshot-00000000685E7DA1-12B74280.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:49,329 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower291: local snapshot saved in 5.488 ms, deleting Pekko-persisted snapshots 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/leader289/TermInfo.properties does not exist 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader289: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/leader289/journal-v1-1.log 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader289: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/leader289/metadata-v1-1.log 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader289: no eligible files found 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - leader289: SnapshotOffer called. 11:16:49,330 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower292/snapshot-00000000685E7DA1-12B74280.v1 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - leader289: initializing from snapshot taken at 2025-06-27T11:16:49.314Z 11:16:49,330 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower292/journal-v1-1.log 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader289: Set currentTerm=1, votedFor=null 11:16:49,330 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower292/metadata-v1-1.log 11:16:49,330 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower292/snapshot-00000000685E7DA1-12B74280.v1 11:16:49,330 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower291: Recovery completed in 6.476 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - leader289: Updated server config: isVoting: true, peers: [PeerInfo [id=follower292, address=pekko://test/user/follower292, votingState=NON_VOTING], PeerInfo [id=follower291, address=pekko://test/user/follower291, votingState=VOTING], PeerInfo [id=follower290, address=pekko://test/user/follower290, votingState=VOTING]] 11:16:49,330 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower292: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:49.314Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/follower292/snapshot-00000000685E7DA1-12B74280.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:49,330 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - follower292: local snapshot saved in 3.552 ms, deleting Pekko-persisted snapshots 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for leader289 in 71.51 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader289: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:49.314Z 11:16:49,330 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader289: support() called 11:16:49,330 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower292: Recovery completed in 4.075 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:49,331 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader289: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/leader289/snapshot-00000000685E7DA1-12B74280.tmp 11:16:49,333 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=1, votedFor=null 11:16:49,333 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower292: Local TermInfo store seeded with TermInfo{term=1} 11:16:49,333 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower292: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6e9892ad 11:16:49,333 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractReplicatedLog - follower292: Moving last applied index from -1 to -1 11:16:49,333 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] JournalRecovery - follower292: journal recovery completedin 334.1 μs with journalIndex=1 11:16:49,333 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader289: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/leader289/snapshot-00000000685E7DA1-12B74280.v1 11:16:49,333 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=1, votedFor=null 11:16:49,333 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower291: Local TermInfo store seeded with TermInfo{term=1} 11:16:49,333 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower291: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@56b34df9 11:16:49,333 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader289: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/leader289/journal-v1-1.log 11:16:49,333 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower291: Moving last applied index from -1 to -1 11:16:49,333 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader289: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/leader289/metadata-v1-1.log 11:16:49,333 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader289: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/leader289/snapshot-00000000685E7DA1-12B74280.v1 11:16:49,333 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader289: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:49.314Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7946513699342950506/leader289/snapshot-00000000685E7DA1-12B74280.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:49,333 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader289: local snapshot saved in 3.308 ms, deleting Pekko-persisted snapshots 11:16:49,334 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - follower291: journal recovery completedin 215.5 μs with journalIndex=1 11:16:49,334 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader289: Recovery completed in 4.431 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:49,335 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower291: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:49,335 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - follower292: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:49,336 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader289: Set currentTerm=1, votedFor=null 11:16:49,336 AM [test-pekko.actor.default-dispatcher-15] [INFO] PekkoRecovery - leader289: Local TermInfo store seeded with TermInfo{term=1} 11:16:49,336 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader289: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@7f0a2c0e 11:16:49,336 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader289: Moving last applied index from -1 to -1 11:16:49,336 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] JournalRecovery - leader289: journal recovery completedin 167.2 μs with journalIndex=1 11:16:49,337 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader289: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:49,730 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:49,730 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - leader289 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.168:17355, Removed) 11:16:49,730 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - leader289 (Follower): Received ElectionTimeout - switching to Candidate 11:16:49,730 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader289 (Candidate): Election: Candidate has following voting peers: [follower291, follower290] 11:16:49,733 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader289: Set currentTerm=2, votedFor=leader289 11:16:49,734 AM [test-pekko.actor.default-dispatcher-15] [INFO] Candidate - leader289 (Candidate): Starting new election term 2 11:16:49,734 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader289 (Candidate): Sending RequestVote{term=2, candidateId=leader289, lastLogIndex=-1, lastLogTerm=-1} to peer follower291 11:16:49,734 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader289 (Candidate): Sending RequestVote{term=2, candidateId=leader289, lastLogIndex=-1, lastLogTerm=-1} to peer follower290 11:16:49,734 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower291 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:49,734 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - leader289 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:16:49,734 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower290 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:49,734 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower291 (Follower): Cluster state: [] 11:16:49,734 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower291 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader289, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:49,734 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower290 (Follower): Cluster state: [] 11:16:49,734 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower291 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader289, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:49,734 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower290 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader289, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:49,734 AM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower290 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader289, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:49,738 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower290: Set currentTerm=2, votedFor=null 11:16:49,738 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower290 (Follower): In requestVote: RequestVote{term=2, candidateId=leader289, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:49,739 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=2, votedFor=null 11:16:49,739 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower291 (Follower): In requestVote: RequestVote{term=2, candidateId=leader289, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:49,741 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower290: Set currentTerm=2, votedFor=leader289 11:16:49,741 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower290 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:49,741 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader289 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 11:16:49,741 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Candidate - leader289 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 11:16:49,741 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader289 (Leader): Election: Leader has following peers: [follower292, follower291, follower290] 11:16:49,741 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower292: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,741 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower291: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,741 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower290: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,741 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - leader289 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:16:49,741 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower292 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader289, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 11:16:49,742 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower290: Last sync leader does not match current leader leader289, need to catch up to -1 11:16:49,742 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader289 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower290 updated: matchIndex: -1, nextIndex: 0 11:16:49,742 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower290: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:49.742] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower290] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower290, initialSyncDone=false] 11:16:49,742 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=2, votedFor=leader289 11:16:49,742 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower291 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:49,742 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower291: Last sync leader does not match current leader leader289, need to catch up to -1 11:16:49,742 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader289 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower291 updated: matchIndex: -1, nextIndex: 0 11:16:49,742 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:49.742] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=false] 11:16:49,744 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=2, votedFor=null 11:16:49,744 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower292: Last sync leader does not match current leader leader289, need to catch up to -1 11:16:49,744 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader289 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower292 updated: matchIndex: -1, nextIndex: 0 11:16:49,744 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:49.744] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower292] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower292, initialSyncDone=false] 11:16:49,790 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:16:49,790 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:16:49,790 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:16:49,790 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:16:49,791 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:16:49,791 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader289: onRequestLeadership RequestLeadership [requestedFollowerId=follower291, replyTo=Actor[pekko://test/user/$a#550415705]] 11:16:49,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader289: Initiating leader transfer 11:16:49,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader289: pauseLeader successfully completed - doing transfer 11:16:49,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Leader - leader289 (Leader): Attempting to transfer leadership 11:16:49,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower290: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower292: Lagging 0 entries behind leader leader289 and reached -1 (of expected -1) 11:16:49,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower292: onLeaderTransitioning: LeaderTransitioning [leaderId=leader289] 11:16:49,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower291: Lagging 0 entries behind leader leader289 and reached -1 (of expected -1) 11:16:49,793 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower291: onLeaderTransitioning: LeaderTransitioning [leaderId=leader289] 11:16:49,794 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower290: Lagging 0 entries behind leader leader289 and reached -1 (of expected -1) 11:16:49,794 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower290: onLeaderTransitioning: LeaderTransitioning [leaderId=leader289] 11:16:49,794 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Leader - leader289 (Leader): tryToCompleteLeadershipTransfer: followerId: follower291, matchIndex: -1, lastIndex: -1, isVoting: true 11:16:49,794 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Leader - leader289 (Leader): Follower's log matches - sending ElectionTimeout 11:16:49,794 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,794 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower291: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,794 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader289 (Leader): Checking sendAppendEntries for follower follower290: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,794 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Leader - leader289 (Leader): Leader transfer complete 11:16:49,794 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader289: leader transfer complete - waiting for new leader [WARN] [06/27/2025 11:16:49.794] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower292] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower292, initialSyncDone=true] [WARN] [06/27/2025 11:16:49.794] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower291] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower291, initialSyncDone=true] [WARN] [06/27/2025 11:16:49.794] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower290] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower290, initialSyncDone=true] 11:16:49,795 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower292: Lagging 0 entries behind leader leader289 and reached -1 (of expected -1) 11:16:49,795 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower291: Lagging 0 entries behind leader leader289 and reached -1 (of expected -1) 11:16:49,795 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:49,795 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower291 (Follower): Received TimeoutNow - switching to Candidate 11:16:49,795 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - follower291 (Candidate): Election: Candidate has following voting peers: [leader289, follower290] 11:16:49,795 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower290: Lagging 0 entries behind leader leader289 and reached -1 (of expected -1) 11:16:49,798 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower291: Set currentTerm=3, votedFor=follower291 11:16:49,798 AM [test-pekko.actor.default-dispatcher-6] [INFO] Candidate - follower291 (Candidate): Starting new election term 3 11:16:49,798 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - follower291 (Candidate): Sending RequestVote{term=3, candidateId=follower291, lastLogIndex=-1, lastLogTerm=-1} to peer leader289 11:16:49,798 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - follower291 (Candidate): Sending RequestVote{term=3, candidateId=follower291, lastLogIndex=-1, lastLogTerm=-1} to peer follower290 11:16:49,798 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - follower291 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 11:16:49,798 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader289 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:49,798 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader289 (Leader): Cluster state: [] 11:16:49,798 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader289 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower291, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:49,798 AM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractLeader - leader289 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower291, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower 11:16:49,799 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower290 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:49,799 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower290 (Follower): Cluster state: [] 11:16:49,799 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower290 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower291, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:49,799 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower290 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower291, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 2 - updating term 11:16:49,801 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader289: Set currentTerm=3, votedFor=null 11:16:49,801 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader289 (Leader): Leadership transfer in progress - processing RequestVote 11:16:49,801 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader289 (Leader): In requestVote: RequestVote{term=3, candidateId=follower291, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:49,802 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower290: Set currentTerm=3, votedFor=null 11:16:49,802 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower290 (Follower): In requestVote: RequestVote{term=3, candidateId=follower291, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:49,804 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader289: Set currentTerm=3, votedFor=follower291 11:16:49,804 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader289 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 11:16:49,804 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - follower291 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 11:16:49,805 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - leader289 (Leader) :- Switching from behavior Leader to Follower, election term: 3 11:16:49,805 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - follower291 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 11:16:49,805 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower291 (Leader): Election: Leader has following peers: [leader289, follower292, follower290] 11:16:49,805 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower291 (Leader): Checking sendAppendEntries for follower leader289: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,805 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower290: Set currentTerm=3, votedFor=follower291 11:16:49,805 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower290 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 11:16:49,805 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - leader289: Last sync leader does not match current leader follower291, need to catch up to -1 11:16:49,806 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader289: leader changed to follower291 11:16:49,806 AM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorLeadershipTransferCohort - leader289: Successfully transferred leadership to follower291 in 12.80 ms 11:16:49,806 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader289: Leadership transferred successfully to follower291 [WARN] [06/27/2025 11:16:49.806] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/leader289] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader289, initialSyncDone=false] 11:16:49,806 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower291 (Leader): Checking sendAppendEntries for follower follower292: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,806 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower291 (Leader): Checking sendAppendEntries for follower follower290: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,806 AM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower292 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower291, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 11:16:49,806 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorBehavior - follower291 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 11:16:49,806 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower291 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader289 updated: matchIndex: -1, nextIndex: 0 11:16:49,806 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower290: Last sync leader does not match current leader follower291, need to catch up to -1 11:16:49,806 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower291 (Leader): Checking sendAppendEntries for follower leader289: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:49.806] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower290] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower290, initialSyncDone=false] 11:16:49,807 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower291 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower290 updated: matchIndex: -1, nextIndex: 0 11:16:49,807 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower291 (Leader): Checking sendAppendEntries for follower follower290: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,808 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower292: Set currentTerm=3, votedFor=null 11:16:49,809 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower292: Last sync leader does not match current leader follower291, need to catch up to -1 [WARN] [06/27/2025 11:16:49.809] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower292] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower292, initialSyncDone=false] 11:16:49,809 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower291 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower292 updated: matchIndex: -1, nextIndex: 0 11:16:49,809 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower291 (Leader): Checking sendAppendEntries for follower follower292: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:49,843 AM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 11:16:49,843 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower290-notifier293#-1167781147] 11:16:49,854 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower290-collector294#-921745068] 11:16:49,854 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower290] 11:16:49,855 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower290: terminating thread VirtualThread[#711,follower290-writer-135]/waiting 11:16:49,855 AM [follower290-writer-135] [DEBUG] JournalWriteTask - follower290: received 1 entries in 537.1 ms 11:16:49,855 AM [follower290-writer-135] [DEBUG] JournalWriteTask - follower290: persisted 1 entries in 26.76 μs 11:16:49,855 AM [follower290-writer-135] [DEBUG] JournalWriteTask - follower290: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@459150db stopped 11:16:49,855 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower290: journal closed 11:16:49,855 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower290: journal closed 11:16:49,855 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower290: stopped executor with 0 remaining tasks 11:16:49,855 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower290: stopped executor with 0 remaining tasks 11:16:49,855 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower291-notifier295#-625912359] 11:16:49,856 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower291-collector296#2038227986] 11:16:49,856 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower291] 11:16:49,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower291: terminating thread VirtualThread[#712,follower291-writer-136]/waiting 11:16:49,856 AM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: received 1 entries in 533.3 ms 11:16:49,856 AM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: persisted 1 entries in 9.150 μs 11:16:49,856 AM [follower291-writer-136] [DEBUG] JournalWriteTask - follower291: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@2bb0d968 stopped 11:16:49,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower291: journal closed 11:16:49,856 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower291: journal closed 11:16:49,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: stopped executor with 0 remaining tasks 11:16:49,856 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower291: stopped executor with 0 remaining tasks 11:16:49,856 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower292-notifier297#600718336] 11:16:49,856 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower292-collector298#1627549922] 11:16:49,857 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower292] 11:16:49,857 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower292: terminating thread VirtualThread[#713,follower292-writer-137]/waiting 11:16:49,857 AM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: received 1 entries in 531.2 ms 11:16:49,857 AM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: persisted 1 entries in 8.850 μs 11:16:49,857 AM [follower292-writer-137] [DEBUG] JournalWriteTask - follower292: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@23d02bbc stopped 11:16:49,857 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower292: journal closed 11:16:49,857 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower292: journal closed 11:16:49,857 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower292: stopped executor with 0 remaining tasks 11:16:49,857 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower292: stopped executor with 0 remaining tasks 11:16:49,857 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader289-notifier299#-1593015521] 11:16:49,857 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader289-collector300#452173570] 11:16:49,858 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader289] 11:16:49,858 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - leader289: terminating thread VirtualThread[#714,leader289-writer-138]/waiting 11:16:49,858 AM [leader289-writer-138] [DEBUG] JournalWriteTask - leader289: received 1 entries in 529.1 ms 11:16:49,858 AM [leader289-writer-138] [DEBUG] JournalWriteTask - leader289: persisted 1 entries in 11.30 μs 11:16:49,858 AM [leader289-writer-138] [DEBUG] JournalWriteTask - leader289: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@4ef71faf stopped 11:16:49,858 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - leader289: journal closed 11:16:49,858 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - leader289: journal closed 11:16:49,858 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader289: stopped executor with 0 remaining tasks 11:16:49,858 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader289: stopped executor with 0 remaining tasks 11:16:49,858 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$a#550415705] 11:16:49,861 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 11:16:49,861 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/27/2025 11:16:49.863] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower302]: org.opendaylight.controller.cluster.common.actor.Monitor@61e83c31 11:16:49,864 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActor - follower302: Starting recovery with journal batch size 1000 11:16:49,864 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: started executor 11:16:49,864 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: started executor 11:16:49,864 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower302: metadata journal open: replayFrom=1 applyTo=0 11:16:49,865 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower302: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:49,865 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower302: journal open: applyTo=0 11:16:49,865 AM [follower302-writer-139] [DEBUG] JournalWriteTask - follower302: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@1af9922 started 11:16:49,865 AM [follower302-writer-139] [DEBUG] JournalWriteTask - follower302: waiting for more work 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower302/TermInfo.properties does not exist 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower302/journal-v1-1.log 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower302/metadata-v1-1.log 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: no eligible files found 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower302: SnapshotOffer called. 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower302: initializing from snapshot taken at 2025-06-27T11:16:49.861Z 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower302: Set currentTerm=1, votedFor=null 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - follower302: Updated server config: isVoting: true, peers: [PeerInfo [id=leader301, address=pekko://test/user/leader301, votingState=VOTING], PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=NON_VOTING], PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING]] 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for follower302 in 70.35 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower302: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:49.861Z 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - follower302: support() called 11:16:49,866 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower302/snapshot-00000000685E7DA1-3351D140.tmp 11:16:49,867 AM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - follower303: Starting recovery with journal batch size 1000 [WARN] [06/27/2025 11:16:49.868] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower303]: org.opendaylight.controller.cluster.common.actor.Monitor@514dcbea 11:16:49,868 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: started executor 11:16:49,868 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: started executor 11:16:49,868 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower303: metadata journal open: replayFrom=1 applyTo=0 11:16:49,869 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower303: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:49,869 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower303: journal open: applyTo=0 11:16:49,869 AM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@2c796be6 started 11:16:49,869 AM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: waiting for more work [WARN] [06/27/2025 11:16:49.870] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower304]: org.opendaylight.controller.cluster.common.actor.Monitor@5f30a01d 11:16:49,871 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - follower304: Starting recovery with journal batch size 1000 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower303/TermInfo.properties does not exist 11:16:49,871 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: started executor 11:16:49,871 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: started executor 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower303/journal-v1-1.log 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower303/metadata-v1-1.log 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: no eligible files found 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower303: SnapshotOffer called. 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRecovery - follower303: initializing from snapshot taken at 2025-06-27T11:16:49.861Z 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=1, votedFor=null 11:16:49,871 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower304: metadata journal open: replayFrom=1 applyTo=0 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PeerInfos - follower303: Updated server config: isVoting: true, peers: [PeerInfo [id=leader301, address=pekko://test/user/leader301, votingState=VOTING], PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=NON_VOTING], PeerInfo [id=follower302, address=pekko://test/user/follower302, votingState=VOTING]] 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - Recovery snapshot applied for follower303 in 85.60 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower303: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:49.861Z 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower303: support() called 11:16:49,871 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower304: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:49,871 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower304: journal open: applyTo=0 11:16:49,871 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower303/snapshot-00000000685E7DA1-3351D140.tmp 11:16:49,871 AM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@ed6281e started 11:16:49,871 AM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: waiting for more work 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower304/TermInfo.properties does not exist 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower304/journal-v1-1.log 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower304/metadata-v1-1.log 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: no eligible files found 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower304: SnapshotOffer called. 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower304: initializing from snapshot taken at 2025-06-27T11:16:49.861Z 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=1, votedFor=null 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower304: Updated server config: isVoting: false, peers: [PeerInfo [id=leader301, address=pekko://test/user/leader301, votingState=VOTING], PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=follower302, address=pekko://test/user/follower302, votingState=VOTING]] 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower304 in 50.27 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower304: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:49.861Z 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower304: support() called 11:16:49,872 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower304/snapshot-00000000685E7DA1-3351D140.tmp [WARN] [06/27/2025 11:16:49.873] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader301]: org.opendaylight.controller.cluster.common.actor.Monitor@5638cc3c 11:16:49,873 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower302/snapshot-00000000685E7DA1-3351D140.v1 11:16:49,874 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower302/snapshot-00000000685E7DA1-3351D140.v1 11:16:49,874 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower302/journal-v1-1.log 11:16:49,874 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower302/metadata-v1-1.log 11:16:49,874 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader301: Starting recovery with journal batch size 1000 11:16:49,874 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower302: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:49.861Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower302/snapshot-00000000685E7DA1-3351D140.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:49,874 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower302: local snapshot saved in 7.616 ms, deleting Pekko-persisted snapshots 11:16:49,874 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: started executor 11:16:49,874 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: started executor 11:16:49,874 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader301: metadata journal open: replayFrom=1 applyTo=0 11:16:49,875 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower302: Recovery completed in 8.652 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:49,875 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader301: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:49,875 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - leader301: journal open: applyTo=0 11:16:49,875 AM [leader301-writer-142] [DEBUG] JournalWriteTask - leader301: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@78aa440c started 11:16:49,875 AM [leader301-writer-142] [DEBUG] JournalWriteTask - leader301: waiting for more work 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/leader301/TermInfo.properties does not exist 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/leader301/journal-v1-1.log 11:16:49,876 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower304/snapshot-00000000685E7DA1-3351D140.v1 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/leader301/metadata-v1-1.log 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: no eligible files found 11:16:49,876 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower304/snapshot-00000000685E7DA1-3351D140.v1 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - leader301: SnapshotOffer called. 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - leader301: initializing from snapshot taken at 2025-06-27T11:16:49.861Z 11:16:49,876 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower304/journal-v1-1.log 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader301: Set currentTerm=1, votedFor=null 11:16:49,876 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower304/metadata-v1-1.log 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - leader301: Updated server config: isVoting: true, peers: [PeerInfo [id=follower304, address=pekko://test/user/follower304, votingState=NON_VOTING], PeerInfo [id=follower303, address=pekko://test/user/follower303, votingState=VOTING], PeerInfo [id=follower302, address=pekko://test/user/follower302, votingState=VOTING]] 11:16:49,876 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:49.861Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower304/snapshot-00000000685E7DA1-3351D140.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:49,876 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower304: local snapshot saved in 3.877 ms, deleting Pekko-persisted snapshots 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for leader301 in 212.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader301: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:49.861Z 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader301: support() called 11:16:49,876 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/leader301/snapshot-00000000685E7DA1-3351D140.tmp 11:16:49,877 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower304: Recovery completed in 4.589 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:49,877 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower302: Set currentTerm=1, votedFor=null 11:16:49,877 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower302: Local TermInfo store seeded with TermInfo{term=1} 11:16:49,877 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower302: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@b7dfe8b 11:16:49,877 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower302: Moving last applied index from -1 to -1 11:16:49,877 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower303/snapshot-00000000685E7DA1-3351D140.v1 11:16:49,877 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower302: journal recovery completedin 328.8 μs with journalIndex=1 11:16:49,877 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower303/snapshot-00000000685E7DA1-3351D140.v1 11:16:49,878 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower303/journal-v1-1.log 11:16:49,878 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower303/metadata-v1-1.log 11:16:49,878 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower303: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:49.861Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/follower303/snapshot-00000000685E7DA1-3351D140.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:49,878 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower303: local snapshot saved in 6.501 ms, deleting Pekko-persisted snapshots 11:16:49,878 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower302: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:49,879 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower303: Recovery completed in 7.585 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:49,879 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/leader301/snapshot-00000000685E7DA1-3351D140.v1 11:16:49,879 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/leader301/snapshot-00000000685E7DA1-3351D140.v1 11:16:49,879 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/leader301/journal-v1-1.log 11:16:49,879 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/leader301/metadata-v1-1.log 11:16:49,879 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader301: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:49.861Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest352460795538362151/leader301/snapshot-00000000685E7DA1-3351D140.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:49,879 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - leader301: local snapshot saved in 2.628 ms, deleting Pekko-persisted snapshots 11:16:49,879 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=1, votedFor=null 11:16:49,879 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower304: Local TermInfo store seeded with TermInfo{term=1} 11:16:49,879 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower304: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@7dc6756c 11:16:49,879 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower304: Moving last applied index from -1 to -1 11:16:49,880 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - leader301: Recovery completed in 3.586 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:49,880 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - follower304: journal recovery completedin 227.1 μs with journalIndex=1 11:16:49,880 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower304: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:49,882 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=1, votedFor=null 11:16:49,882 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower303: Local TermInfo store seeded with TermInfo{term=1} 11:16:49,882 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower303: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6f12ed15 11:16:49,882 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower303: Moving last applied index from -1 to -1 11:16:49,882 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower303: journal recovery completedin 243.3 μs with journalIndex=1 11:16:49,883 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - leader301: Set currentTerm=1, votedFor=null 11:16:49,883 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - leader301: Local TermInfo store seeded with TermInfo{term=1} 11:16:49,883 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader301: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@5dda5b3b 11:16:49,883 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - leader301: Moving last applied index from -1 to -1 11:16:49,883 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower303: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:49,883 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - leader301: journal recovery completedin 287.0 μs with journalIndex=1 11:16:49,884 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - leader301: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:50,230 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:50,231 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - leader301 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.168:17355, Removed) 11:16:50,231 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - leader301 (Follower): Received ElectionTimeout - switching to Candidate 11:16:50,231 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader301 (Candidate): Election: Candidate has following voting peers: [follower303, follower302] 11:16:50,235 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader301: Set currentTerm=2, votedFor=leader301 11:16:50,235 AM [test-pekko.actor.default-dispatcher-17] [INFO] Candidate - leader301 (Candidate): Starting new election term 2 11:16:50,235 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader301 (Candidate): Sending RequestVote{term=2, candidateId=leader301, lastLogIndex=-1, lastLogTerm=-1} to peer follower303 11:16:50,235 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader301 (Candidate): Sending RequestVote{term=2, candidateId=leader301, lastLogIndex=-1, lastLogTerm=-1} to peer follower302 11:16:50,235 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorBehavior - leader301 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:16:50,235 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower303 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:50,235 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower302 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:50,235 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower303 (Follower): Cluster state: [] 11:16:50,235 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower302 (Follower): Cluster state: [] 11:16:50,235 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower303 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader301, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:50,235 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower302 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader301, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:50,235 AM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower303 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader301, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:50,235 AM [test-pekko.actor.default-dispatcher-7] [INFO] Follower - follower302 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader301, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:50,238 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=2, votedFor=null 11:16:50,238 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower303 (Follower): In requestVote: RequestVote{term=2, candidateId=leader301, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:50,239 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower302: Set currentTerm=2, votedFor=null 11:16:50,239 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower302 (Follower): In requestVote: RequestVote{term=2, candidateId=leader301, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:50,242 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower303: Set currentTerm=2, votedFor=leader301 11:16:50,242 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower303 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:50,242 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader301 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 11:16:50,242 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - leader301 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 11:16:50,242 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader301 (Leader): Election: Leader has following peers: [follower304, follower303, follower302] 11:16:50,242 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader301 (Leader): Checking sendAppendEntries for follower follower304: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:50,242 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader301 (Leader): Checking sendAppendEntries for follower follower303: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:50,242 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader301 (Leader): Checking sendAppendEntries for follower follower302: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:50,242 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower304 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader301, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 11:16:50,242 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower302: Set currentTerm=2, votedFor=leader301 11:16:50,242 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActorBehavior - follower302 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:50,243 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower303: Last sync leader does not match current leader leader301, need to catch up to -1 [WARN] [06/27/2025 11:16:50.243] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower303] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower303, initialSyncDone=false] 11:16:50,243 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower302: Last sync leader does not match current leader leader301, need to catch up to -1 [WARN] [06/27/2025 11:16:50.243] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower302] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower302, initialSyncDone=false] 11:16:50,243 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorBehavior - leader301 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:16:50,243 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader301 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower303 updated: matchIndex: -1, nextIndex: 0 11:16:50,243 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader301 (Leader): Checking sendAppendEntries for follower follower303: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:50,244 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader301 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower302 updated: matchIndex: -1, nextIndex: 0 11:16:50,244 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader301 (Leader): Checking sendAppendEntries for follower follower302: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:50,245 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower304: Set currentTerm=2, votedFor=null 11:16:50,245 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower304: Last sync leader does not match current leader leader301, need to catch up to -1 11:16:50,245 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader301 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower304 updated: matchIndex: -1, nextIndex: 0 11:16:50,245 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader301 (Leader): Checking sendAppendEntries for follower follower304: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:50.245] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower304] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower304, initialSyncDone=false] 11:16:50,286 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:16:50,286 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:16:50,287 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:16:50,287 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower303 starting 11:16:50,287 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower303: onShutDown 11:16:50,287 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRaftStorage - follower303: terminating thread VirtualThread[#716,follower303-writer-140]/waiting 11:16:50,287 AM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: received 1 entries in 418.3 ms 11:16:50,287 AM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: persisted 1 entries in 33.73 μs 11:16:50,287 AM [follower303-writer-140] [DEBUG] JournalWriteTask - follower303: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@2c796be6 stopped 11:16:50,287 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower303: journal closed 11:16:50,287 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - follower303: journal closed 11:16:50,287 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower303: stopped executor with 0 remaining tasks 11:16:50,287 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower303: stopped executor with 0 remaining tasks 11:16:50,288 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower303 ending 11:16:50,288 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:16:50,288 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:16:50,288 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader301: onRequestLeadership RequestLeadership [requestedFollowerId=follower303, replyTo=Actor[pekko://test/user/$b#-1102586478]] 11:16:50,288 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader301: Initiating leader transfer 11:16:50,288 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower304: onLeaderTransitioning: LeaderTransitioning [leaderId=leader301] [WARN] [06/27/2025 11:16:50.288] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower303] received dead letter from TestActor[pekko://test/user/leader301]: LeaderTransitioning [leaderId=leader301] 11:16:50,288 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower302: onLeaderTransitioning: LeaderTransitioning [leaderId=leader301] 11:16:50,288 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader301: pauseLeader successfully completed - doing transfer 11:16:50,288 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Leader - leader301 (Leader): Attempting to transfer leadership 11:16:50,288 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader301 (Leader): Checking sendAppendEntries for follower follower304: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:50,288 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader301 (Leader): Checking sendAppendEntries for follower follower303: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:50,288 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower304: Lagging 0 entries behind leader leader301 and reached -1 (of expected -1) 11:16:50,288 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader301 (Leader): Checking sendAppendEntries for follower follower302: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:50.288] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower303] received dead letter from TestActor[pekko://test/user/leader301]: AppendEntries{term=2, leaderId=leader301, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 11:16:50,289 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower302: Lagging 0 entries behind leader leader301 and reached -1 (of expected -1) [WARN] [06/27/2025 11:16:50.289] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower304] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower304, initialSyncDone=true] [WARN] [06/27/2025 11:16:50.289] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower302] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower302, initialSyncDone=true] [WARN] [06/27/2025 11:16:50.360] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower303] received dead letter from TestActor[pekko://test/user/leader301]: AppendEntries{term=2, leaderId=leader301, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 11:16:50,480 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower304: Lagging 0 entries behind leader leader301 and reached -1 (of expected -1) [WARN] [06/27/2025 11:16:50.480] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower303] received dead letter from TestActor[pekko://test/user/leader301]: AppendEntries{term=2, leaderId=leader301, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 11:16:50,480 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower302: Lagging 0 entries behind leader leader301 and reached -1 (of expected -1) 11:16:50,600 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader301: leader transfer aborted 11:16:50,600 AM [test-pekko.actor.default-dispatcher-12] [WARN] RaftActorLeadershipTransferCohort - leader301: Failed to transfer leadership in 312.2 ms 11:16:50,600 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader301: LeadershipTransfer request from follower303 failed 11:16:50,601 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Leader - leader301 (Leader): Leadership transfer expired 11:16:50,601 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower304: Lagging 0 entries behind leader leader301 and reached -1 (of expected -1) [WARN] [06/27/2025 11:16:50.601] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower303] received dead letter from TestActor[pekko://test/user/leader301]: AppendEntries{term=2, leaderId=leader301, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 11:16:50,601 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower302: Lagging 0 entries behind leader leader301 and reached -1 (of expected -1) 11:16:50,640 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 11:16:50,640 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower302-notifier305#-233375636] 11:16:50,651 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower302-collector306#-2046154493] 11:16:50,651 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower302] 11:16:50,652 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower302: terminating thread VirtualThread[#715,follower302-writer-139]/waiting 11:16:50,652 AM [follower302-writer-139] [DEBUG] JournalWriteTask - follower302: received 1 entries in 786.6 ms 11:16:50,652 AM [follower302-writer-139] [DEBUG] JournalWriteTask - follower302: persisted 1 entries in 33.07 μs 11:16:50,652 AM [follower302-writer-139] [DEBUG] JournalWriteTask - follower302: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@1af9922 stopped 11:16:50,652 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower302: journal closed 11:16:50,652 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower302: journal closed 11:16:50,652 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower302: stopped executor with 0 remaining tasks 11:16:50,652 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower302: stopped executor with 0 remaining tasks 11:16:50,652 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower303-notifier307#-2004638994] 11:16:50,652 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower303-collector308#-1893429781] 11:16:50,653 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower303] 11:16:50,653 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower304-notifier309#751383198] 11:16:50,653 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower304-collector310#-564787569] 11:16:50,653 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower304] 11:16:50,653 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower304: terminating thread VirtualThread[#717,follower304-writer-141]/waiting 11:16:50,653 AM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: received 1 entries in 782.0 ms 11:16:50,654 AM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: persisted 1 entries in 20.80 μs 11:16:50,654 AM [follower304-writer-141] [DEBUG] JournalWriteTask - follower304: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@ed6281e stopped 11:16:50,654 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower304: journal closed 11:16:50,654 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower304: journal closed 11:16:50,654 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: stopped executor with 0 remaining tasks 11:16:50,654 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower304: stopped executor with 0 remaining tasks 11:16:50,654 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader301-notifier311#-1951363547] 11:16:50,654 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader301-collector312#907565169] 11:16:50,654 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader301] 11:16:50,654 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - leader301: terminating thread VirtualThread[#718,leader301-writer-142]/waiting 11:16:50,654 AM [leader301-writer-142] [DEBUG] JournalWriteTask - leader301: received 1 entries in 779.3 ms 11:16:50,655 AM [leader301-writer-142] [DEBUG] JournalWriteTask - leader301: persisted 1 entries in 19.01 μs 11:16:50,655 AM [leader301-writer-142] [DEBUG] JournalWriteTask - leader301: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@78aa440c stopped 11:16:50,655 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader301: journal closed 11:16:50,655 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader301: journal closed 11:16:50,655 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader301: stopped executor with 0 remaining tasks 11:16:50,655 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader301: stopped executor with 0 remaining tasks 11:16:50,655 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$b#-1102586478] 11:16:50,658 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 11:16:50,658 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/27/2025 11:16:50.660] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower314]: org.opendaylight.controller.cluster.common.actor.Monitor@18209578 11:16:50,660 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - follower314: Starting recovery with journal batch size 1000 11:16:50,660 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: started executor 11:16:50,660 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: started executor 11:16:50,661 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower314: metadata journal open: replayFrom=1 applyTo=0 11:16:50,661 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower314: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:50,661 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower314: journal open: applyTo=0 11:16:50,661 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@531a4b2 started 11:16:50,661 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: waiting for more work 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower314/TermInfo.properties does not exist 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower314/journal-v1-1.log 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower314/metadata-v1-1.log 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: no eligible files found 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower314: SnapshotOffer called. 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower314: initializing from snapshot taken at 2025-06-27T11:16:50.658Z 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower314: Set currentTerm=1, votedFor=null 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower314: Updated server config: isVoting: true, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=leader313, address=pekko://test/user/leader313, votingState=VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=NON_VOTING]] 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower314 in 79.58 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower314: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:50.658Z 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower314: support() called 11:16:50,662 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower314/snapshot-00000000685E7DA2-27384880.tmp [WARN] [06/27/2025 11:16:50.663] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower315]: org.opendaylight.controller.cluster.common.actor.Monitor@486b813 11:16:50,663 AM [test-pekko.actor.default-dispatcher-7] [INFO] RaftActor - follower315: Starting recovery with journal batch size 1000 11:16:50,663 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: started executor 11:16:50,663 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: started executor 11:16:50,663 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower315: metadata journal open: replayFrom=1 applyTo=0 11:16:50,664 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] EntryJournalV1 - follower315: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:50,664 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRaftStorage - follower315: journal open: applyTo=0 11:16:50,664 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@62f0c9a9 started 11:16:50,664 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: waiting for more work 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower315/TermInfo.properties does not exist 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower315/journal-v1-1.log 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower315/metadata-v1-1.log 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: no eligible files found 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower315: SnapshotOffer called. 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PekkoRecovery - follower315: initializing from snapshot taken at 2025-06-27T11:16:50.658Z 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=1, votedFor=null 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PeerInfos - follower315: Updated server config: isVoting: true, peers: [PeerInfo [id=follower314, address=pekko://test/user/follower314, votingState=VOTING], PeerInfo [id=leader313, address=pekko://test/user/leader313, votingState=VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=NON_VOTING]] 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - Recovery snapshot applied for follower315 in 103.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower315: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:50.658Z 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [INFO] MockRaftActor - follower315: support() called 11:16:50,665 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower315/snapshot-00000000685E7DA2-27384880.tmp [WARN] [06/27/2025 11:16:50.668] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower316]: org.opendaylight.controller.cluster.common.actor.Monitor@3c53d309 11:16:50,668 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower316: Starting recovery with journal batch size 1000 11:16:50,668 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: started executor 11:16:50,668 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: started executor 11:16:50,669 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower314/snapshot-00000000685E7DA2-27384880.v1 11:16:50,669 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower314/snapshot-00000000685E7DA2-27384880.v1 11:16:50,669 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower314/journal-v1-1.log 11:16:50,669 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower314/metadata-v1-1.log 11:16:50,669 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:50.658Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower314/snapshot-00000000685E7DA2-27384880.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:50,669 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower314: local snapshot saved in 7.084 ms, deleting Pekko-persisted snapshots 11:16:50,670 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower315/snapshot-00000000685E7DA2-27384880.v1 11:16:50,670 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower315/snapshot-00000000685E7DA2-27384880.v1 11:16:50,670 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower316: metadata journal open: replayFrom=1 applyTo=0 11:16:50,670 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower315/journal-v1-1.log 11:16:50,670 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower315/metadata-v1-1.log 11:16:50,670 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftStorage - follower315: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:50.658Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower315/snapshot-00000000685E7DA2-27384880.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:50,670 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower315: local snapshot saved in 4.787 ms, deleting Pekko-persisted snapshots 11:16:50,670 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower314: Recovery completed in 8.039 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:50,670 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower315: Recovery completed in 5.380 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:50,670 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower316: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:50,670 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - follower316: journal open: applyTo=0 11:16:50,670 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@5dd98b31 started 11:16:50,671 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: waiting for more work [WARN] [06/27/2025 11:16:50.672] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader313]: org.opendaylight.controller.cluster.common.actor.Monitor@363d413d 11:16:50,672 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower316/TermInfo.properties does not exist 11:16:50,672 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower316/journal-v1-1.log 11:16:50,672 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower314: Set currentTerm=1, votedFor=null 11:16:50,672 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower316/metadata-v1-1.log 11:16:50,672 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower314: Local TermInfo store seeded with TermInfo{term=1} 11:16:50,672 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower314: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@27bf402a 11:16:50,672 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: no eligible files found 11:16:50,672 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower314: Moving last applied index from -1 to -1 11:16:50,672 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower316: SnapshotOffer called. 11:16:50,672 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower316: initializing from snapshot taken at 2025-06-27T11:16:50.658Z 11:16:50,673 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=1, votedFor=null 11:16:50,673 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower316: Updated server config: isVoting: false, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=follower314, address=pekko://test/user/follower314, votingState=VOTING], PeerInfo [id=leader313, address=pekko://test/user/leader313, votingState=VOTING]] 11:16:50,673 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for follower316 in 91.55 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:50,673 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower316: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:50.658Z 11:16:50,673 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower316: support() called 11:16:50,673 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower314: journal recovery completedin 307.6 μs with journalIndex=1 11:16:50,673 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower316/snapshot-00000000685E7DA2-27384880.tmp 11:16:50,673 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower314: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:50,673 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=1, votedFor=null 11:16:50,673 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower315: Local TermInfo store seeded with TermInfo{term=1} 11:16:50,673 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower315: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@23cfb158 11:16:50,673 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower315: Moving last applied index from -1 to -1 11:16:50,673 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower315: journal recovery completedin 197.5 μs with journalIndex=1 11:16:50,674 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - leader313: Starting recovery with journal batch size 1000 11:16:50,674 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: started executor 11:16:50,674 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: started executor 11:16:50,674 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower315: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:50,674 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader313: metadata journal open: replayFrom=1 applyTo=0 11:16:50,674 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader313: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:50,674 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader313: journal open: applyTo=0 11:16:50,674 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@77793ea2 started 11:16:50,674 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: waiting for more work 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/leader313/TermInfo.properties does not exist 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/leader313/journal-v1-1.log 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/leader313/metadata-v1-1.log 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: no eligible files found 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - leader313: SnapshotOffer called. 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - leader313: initializing from snapshot taken at 2025-06-27T11:16:50.658Z 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader313: Set currentTerm=1, votedFor=null 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - leader313: Updated server config: isVoting: true, peers: [PeerInfo [id=follower315, address=pekko://test/user/follower315, votingState=VOTING], PeerInfo [id=follower314, address=pekko://test/user/follower314, votingState=VOTING], PeerInfo [id=follower316, address=pekko://test/user/follower316, votingState=NON_VOTING]] 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for leader313 in 48.96 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader313: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:50.658Z 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader313: support() called 11:16:50,675 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/leader313/snapshot-00000000685E7DA2-27384880.tmp 11:16:50,676 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower316/snapshot-00000000685E7DA2-27384880.v1 11:16:50,676 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower316/snapshot-00000000685E7DA2-27384880.v1 11:16:50,676 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower316/journal-v1-1.log 11:16:50,676 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower316/metadata-v1-1.log 11:16:50,676 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower316: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:50.658Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/follower316/snapshot-00000000685E7DA2-27384880.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:50,676 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower316: local snapshot saved in 3.166 ms, deleting Pekko-persisted snapshots 11:16:50,677 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower316: Recovery completed in 4.031 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:50,678 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/leader313/snapshot-00000000685E7DA2-27384880.v1 11:16:50,678 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/leader313/snapshot-00000000685E7DA2-27384880.v1 11:16:50,678 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/leader313/journal-v1-1.log 11:16:50,678 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/leader313/metadata-v1-1.log 11:16:50,678 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:50.658Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest15299227882619600881/leader313/snapshot-00000000685E7DA2-27384880.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:50,678 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader313: local snapshot saved in 2.873 ms, deleting Pekko-persisted snapshots 11:16:50,679 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader313: Recovery completed in 3.646 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:50,679 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=1, votedFor=null 11:16:50,679 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower316: Local TermInfo store seeded with TermInfo{term=1} 11:16:50,679 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower316: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@6b52d405 11:16:50,679 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower316: Moving last applied index from -1 to -1 11:16:50,680 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower316: journal recovery completedin 711.5 μs with journalIndex=1 11:16:50,680 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower316: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:50,681 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - leader313: Set currentTerm=1, votedFor=null 11:16:50,681 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - leader313: Local TermInfo store seeded with TermInfo{term=1} 11:16:50,681 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader313: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@3066186b 11:16:50,681 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader313: Moving last applied index from -1 to -1 11:16:50,681 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - leader313: journal recovery completedin 186.2 μs with journalIndex=1 11:16:50,681 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader313: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:51,99 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:51,100 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader313 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.168:17355, Removed) 11:16:51,100 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader313 (Follower): Received ElectionTimeout - switching to Candidate 11:16:51,100 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader313 (Candidate): Election: Candidate has following voting peers: [follower315, follower314] 11:16:51,103 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader313: Set currentTerm=2, votedFor=leader313 11:16:51,104 AM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - leader313 (Candidate): Starting new election term 2 11:16:51,104 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader313 (Candidate): Sending RequestVote{term=2, candidateId=leader313, lastLogIndex=-1, lastLogTerm=-1} to peer follower315 11:16:51,104 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader313 (Candidate): Sending RequestVote{term=2, candidateId=leader313, lastLogIndex=-1, lastLogTerm=-1} to peer follower314 11:16:51,104 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower315 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:51,104 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader313 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:16:51,104 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower314 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:51,104 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower315 (Follower): Cluster state: [] 11:16:51,104 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower315 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader313, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:51,104 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower314 (Follower): Cluster state: [] 11:16:51,104 AM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower315 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader313, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:51,104 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower314 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader313, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:51,104 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower314 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader313, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:51,107 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower314: Set currentTerm=2, votedFor=null 11:16:51,107 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower314 (Follower): In requestVote: RequestVote{term=2, candidateId=leader313, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:51,108 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=2, votedFor=null 11:16:51,109 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower315 (Follower): In requestVote: RequestVote{term=2, candidateId=leader313, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:51,111 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower314: Set currentTerm=2, votedFor=leader313 11:16:51,111 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower314 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:51,111 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader313 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 11:16:51,111 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - leader313 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 11:16:51,111 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): Election: Leader has following peers: [follower315, follower314, follower316] 11:16:51,111 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower315: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:51,111 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower314: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:51,111 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower316: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:51,111 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - leader313 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:16:51,111 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower316 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader313, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 11:16:51,112 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower314 updated: matchIndex: -1, nextIndex: 0 11:16:51,112 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=2, votedFor=leader313 11:16:51,112 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower315 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:51,112 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower314: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:51,112 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower315: Last sync leader does not match current leader leader313, need to catch up to -1 11:16:51,112 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower315 updated: matchIndex: -1, nextIndex: 0 11:16:51,112 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:51,112 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower314: Last sync leader does not match current leader leader313, need to catch up to -1 [WARN] [06/27/2025 11:16:51.112] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower315] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower315, initialSyncDone=false] [WARN] [06/27/2025 11:16:51.112] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower314] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower314, initialSyncDone=false] 11:16:51,114 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=2, votedFor=null 11:16:51,114 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower316: Last sync leader does not match current leader leader313, need to catch up to -1 11:16:51,115 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower316 updated: matchIndex: -1, nextIndex: 0 [WARN] [06/27/2025 11:16:51.115] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=false] 11:16:51,115 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:51,133 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:16:51,133 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader313: Persist data index=0 term=2 command=zero 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - leader313: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,134 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: received 1 entries in 459.2 ms 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): sendAppendEntries: 0 is present for follower follower315 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): Sending AppendEntries to follower follower315: AppendEntries{term=2, leaderId=leader313, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower314: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): sendAppendEntries: 0 is present for follower follower314 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): Sending AppendEntries to follower follower314: AppendEntries{term=2, leaderId=leader313, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): sendAppendEntries: 0 is present for follower follower316 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): Sending AppendEntries to follower follower316: AppendEntries{term=2, leaderId=leader313, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower314 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader313, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower314 (Follower): Number of entries to be appended = 1 11:16:51,134 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower316 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader313, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower314 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:16:51,134 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower316 (Follower): Number of entries to be appended = 1 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower314 (Follower): Append entry to log zero 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower314: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 11:16:51,134 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower316 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:16:51,134 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower316 (Follower): Append entry to log zero 11:16:51,134 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower316: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower314 (Follower): Log size is now 1 11:16:51,134 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower314 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower314, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 11:16:51,134 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower316 (Follower): Log size is now 1 11:16:51,134 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower316 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower316, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 11:16:51,134 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower316: Lagging 0 entries behind leader leader313 and reached -1 (of expected -1) 11:16:51,134 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower314 updated: matchIndex: 0, nextIndex: 1 11:16:51,134 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryStoreCompleter - follower316: awaiting more completions to resolve 1 deferred callback(s) 11:16:51,134 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower314: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,134 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: received 1 entries in 473.2 ms 11:16:51,135 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower316 updated: matchIndex: 0, nextIndex: 1 11:16:51,135 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: received 1 entries in 464.0 ms 11:16:51,135 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,135 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower314: Lagging 0 entries behind leader leader313 and reached -1 (of expected -1) 11:16:51,135 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryStoreCompleter - follower314: awaiting more completions to resolve 1 deferred callback(s) 11:16:51,139 AM [follower314-writer-143] [DEBUG] EntryStoreCompleter - follower314: 1 completion(s) pending 11:16:51,139 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: persisted 1 entries in 4.364 ms 11:16:51,139 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: waiting for more work 11:16:51,139 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryStoreCompleter - follower314: running 1 completion(s) 11:16:51,139 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower314: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 11:16:51,139 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryStoreCompleter - follower314: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c087b3c8@40d4d682 [WARN] [06/27/2025 11:16:51.139] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower314] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower314, initialSyncDone=true] 11:16:51,140 AM [follower316-writer-145] [DEBUG] EntryStoreCompleter - follower316: 1 completion(s) pending 11:16:51,140 AM [leader313-writer-146] [DEBUG] EntryStoreCompleter - leader313: 1 completion(s) pending 11:16:51,140 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: persisted 1 entries in 5.258 ms 11:16:51,140 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: waiting for more work 11:16:51,140 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: persisted 1 entries in 6.083 ms 11:16:51,140 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryStoreCompleter - leader313: running 1 completion(s) 11:16:51,140 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: waiting for more work 11:16:51,140 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader313: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} 11:16:51,140 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryStoreCompleter - follower316: running 1 completion(s) 11:16:51,140 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower316: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 11:16:51,140 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryStoreCompleter - follower316: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c087b3c8@305975c6 11:16:51,140 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader313 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 11:16:51,140 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader313 (Leader): Setting last applied to 0 11:16:51,140 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader313: Moving last applied index from -1 to 0 11:16:51,140 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader313: Applying state for log index 0 data zero 11:16:51,140 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader313: applyState called: zero [WARN] [06/27/2025 11:16:51.140] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=true] 11:16:51,140 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: received 1 entries in 362.6 μs 11:16:51,140 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: persisted 1 entries in 126.7 μs 11:16:51,140 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: waiting for more work 11:16:51,230 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader313 (Leader): sendAppendEntries: 0 is present for follower follower315 11:16:51,230 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader313 (Leader): Sending AppendEntries to follower follower315: AppendEntries{term=2, leaderId=leader313, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 11:16:51,350 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader313 (Leader): sendAppendEntries: 0 is present for follower follower315 11:16:51,350 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader313 (Leader): Sending AppendEntries to follower follower315: AppendEntries{term=2, leaderId=leader313, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 11:16:51,350 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower314 (Follower): Commit index set to 0 11:16:51,350 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower316 (Follower): Commit index set to 0 11:16:51,350 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower314: Lagging 0 entries behind leader leader313 and reached 0 (of expected -1) 11:16:51,350 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower316: Lagging 0 entries behind leader leader313 and reached 0 (of expected -1) 11:16:51,350 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower314 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:16:51,350 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower316 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:16:51,350 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower314 (Follower): Setting last applied to 0 11:16:51,350 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower316 (Follower): Setting last applied to 0 11:16:51,350 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower314: Moving last applied index from -1 to 0 11:16:51,350 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower316: Moving last applied index from -1 to 0 11:16:51,350 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower314: Applying state for log index 0 data zero 11:16:51,350 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower316: Applying state for log index 0 data zero 11:16:51,350 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower314: applyState called: zero 11:16:51,350 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower316: applyState called: zero 11:16:51,350 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: received 1 entries in 211.6 ms 11:16:51,351 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: received 1 entries in 210.7 ms 11:16:51,351 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: persisted 1 entries in 115.3 μs 11:16:51,351 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: waiting for more work 11:16:51,351 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: persisted 1 entries in 118.9 μs 11:16:51,351 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: waiting for more work 11:16:51,385 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 11:16:51,385 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 11:16:51,385 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader313: onShutDown 11:16:51,385 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader313: Initiating leader transfer 11:16:51,386 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorLeadershipTransferCohort - leader313: pauseLeader successfully completed - doing transfer 11:16:51,386 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower316: onLeaderTransitioning: LeaderTransitioning [leaderId=leader313] 11:16:51,386 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] Leader - leader313 (Leader): Attempting to transfer leadership 11:16:51,386 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,386 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader313 (Leader): sendAppendEntries: 0 is present for follower follower315 11:16:51,386 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower314: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,386 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower314: onLeaderTransitioning: LeaderTransitioning [leaderId=leader313] 11:16:51,386 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,386 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower316: Lagging 0 entries behind leader leader313 and reached 0 (of expected -1) 11:16:51,386 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower314: Lagging 0 entries behind leader leader313 and reached 0 (of expected -1) 11:16:51,386 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Leader - leader313 (Leader): tryToCompleteLeadershipTransfer: followerId: follower316, matchIndex: 0, lastIndex: 0, isVoting: false 11:16:51,386 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Leader - leader313 (Leader): tryToCompleteLeadershipTransfer: followerId: follower314, matchIndex: 0, lastIndex: 0, isVoting: true 11:16:51,386 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Leader - leader313 (Leader): Follower's log matches - sending ElectionTimeout 11:16:51,386 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,386 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader313 (Leader): sendAppendEntries: 0 is present for follower follower315 11:16:51,386 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower314: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,386 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader313 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,386 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower314: Lagging 0 entries behind leader leader313 and reached 0 (of expected -1) 11:16:51,386 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Leader - leader313 (Leader): Leader transfer complete 11:16:51,386 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader313: leader transfer complete - waiting for new leader 11:16:51,386 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower316: Lagging 0 entries behind leader leader313 and reached 0 (of expected -1) 11:16:51,386 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:51,386 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower314 (Follower): Received TimeoutNow - switching to Candidate 11:16:51,387 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower314 (Candidate): Election: Candidate has following voting peers: [follower315, leader313] 11:16:51,412 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower314: Set currentTerm=3, votedFor=follower314 11:16:51,412 AM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - follower314 (Candidate): Starting new election term 3 11:16:51,412 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower314 (Candidate): Sending RequestVote{term=3, candidateId=follower314, lastLogIndex=0, lastLogTerm=2} to peer follower315 11:16:51,412 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower314 (Candidate): Sending RequestVote{term=3, candidateId=follower314, lastLogIndex=0, lastLogTerm=2} to peer leader313 11:16:51,412 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - follower314 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 11:16:51,413 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower315 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:51,413 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower315 (Follower): Cluster state: [] 11:16:51,413 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower315 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower314, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 11:16:51,413 AM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower315 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower314, lastLogIndex=0, lastLogTerm=2}" message is greater than follower's term 2 - updating term 11:16:51,413 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - leader313 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:51,413 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - leader313 (Leader): Cluster state: [] 11:16:51,413 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - leader313 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower314, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 11:16:51,413 AM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractLeader - leader313 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower314, lastLogIndex=0, lastLogTerm=2}" message is greater than leader's term 2 - switching to Follower 11:16:51,416 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=3, votedFor=null 11:16:51,416 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower315 (Follower): In requestVote: RequestVote{term=3, candidateId=follower314, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:51,417 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader313: Set currentTerm=3, votedFor=null 11:16:51,417 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader313 (Leader): Leadership transfer in progress - processing RequestVote 11:16:51,417 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - leader313 (Leader): In requestVote: RequestVote{term=3, candidateId=follower314, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 11:16:51,419 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower315: Set currentTerm=3, votedFor=follower314 11:16:51,419 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower315 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 11:16:51,419 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower314 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 11:16:51,419 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower314 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 11:16:51,419 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower314 (Leader): Election: Leader has following peers: [follower315, leader313, follower316] 11:16:51,420 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower314 (Leader): Checking sendAppendEntries for follower follower315: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,420 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower314 (Leader): Checking sendAppendEntries for follower leader313: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,420 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader313: Set currentTerm=3, votedFor=follower314 11:16:51,420 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - follower314 (Leader): Checking sendAppendEntries for follower follower316: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,420 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - leader313 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 11:16:51,420 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - follower314 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 11:16:51,420 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorBehavior - leader313 (Leader) :- Switching from behavior Leader to Follower, election term: 3 11:16:51,420 AM [test-pekko.actor.default-dispatcher-6] [INFO] Follower - follower316 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 11:16:51,420 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - follower314 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader313 updated: matchIndex: 0, nextIndex: 1 11:16:51,420 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - follower314 (Leader): Checking sendAppendEntries for follower leader313: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,420 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - leader313: Last sync leader does not match current leader follower314, need to catch up to -1 11:16:51,421 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader313: leader changed to follower314 11:16:51,421 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorLeadershipTransferCohort - leader313: Successfully transferred leadership to follower314 in 35.04 ms 11:16:51,421 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader313: leader transfer succeeded - sending PoisonPill 11:16:51,421 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - leader313: Lagging 0 entries behind leader follower314 and reached 0 (of expected -1) [WARN] [06/27/2025 11:16:51.421] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/leader313] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader313, initialSyncDone=false] 11:16:51,421 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - leader313: terminating thread VirtualThread[#722,leader313-writer-146]/waiting 11:16:51,421 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: received 1 entries in 280.9 ms 11:16:51,421 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: persisted 1 entries in 27.19 μs 11:16:51,421 AM [leader313-writer-146] [DEBUG] JournalWriteTask - leader313: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@77793ea2 stopped 11:16:51,422 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader313: journal closed 11:16:51,422 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader313: journal closed 11:16:51,422 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: stopped executor with 0 remaining tasks 11:16:51,422 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader313: stopped executor with 0 remaining tasks [WARN] [06/27/2025 11:16:51.422] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/leader313] received dead letter: FollowerInitialSyncUpStatus[memberId=leader313, initialSyncDone=true] 11:16:51,423 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower316: Set currentTerm=3, votedFor=null 11:16:51,423 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower316: Last sync leader does not match current leader follower314, need to catch up to -1 11:16:51,423 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower316 updated: matchIndex: 0, nextIndex: 1 11:16:51,423 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower314 (Leader): Checking sendAppendEntries for follower follower316: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:51.423] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=false] 11:16:51,423 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower316: Lagging 0 entries behind leader follower314 and reached 0 (of expected -1) [WARN] [06/27/2025 11:16:51.423] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower316] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower316, initialSyncDone=true] 11:16:51,931 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower315: Last sync leader does not match current leader follower314, need to catch up to -1 11:16:51,931 AM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower316: Lagging 0 entries behind leader follower314 and reached 0 (of expected -1) 11:16:51,931 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower314 (Leader): sendAppendEntries: 0 is present for follower follower315 11:16:51,931 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - follower314 (Leader): Sending AppendEntries to follower follower315: AppendEntries{term=3, leaderId=follower314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} [WARN] [06/27/2025 11:16:51.931] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/leader313] received dead letter from TestActor[pekko://test/user/follower314]: AppendEntries{term=3, leaderId=follower314, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [06/27/2025 11:16:51.931] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower315] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower315, initialSyncDone=false] 11:16:51,931 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower315 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=follower314, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 11:16:51,931 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower315 (Follower): Number of entries to be appended = 1 11:16:51,931 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower315 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:16:51,931 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower315 (Follower): Append entry to log zero 11:16:51,931 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower315: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=false} 11:16:51,931 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: received 1 entries in 1.267 s 11:16:51,932 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower315 (Follower): Log size is now 1 11:16:51,932 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower315 (Follower): Commit index set to 0 11:16:51,932 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower315 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower315, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 11:16:51,932 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower315: Lagging 0 entries behind leader follower314 and reached 0 (of expected -1) 11:16:51,932 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Follower - follower315 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:16:51,932 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower315 (Follower): Setting last applied to 0 11:16:51,932 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - follower315: Moving last applied index from -1 to 0 11:16:51,932 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower315: Applying state for log index 0 data zero 11:16:51,932 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower315: applyState called: zero 11:16:51,932 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower314 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower315 updated: matchIndex: 0, nextIndex: 1 11:16:51,932 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - follower314 (Leader): Checking sendAppendEntries for follower follower315: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:51,932 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryStoreCompleter - follower315: awaiting more completions to resolve 1 deferred callback(s) 11:16:51,936 AM [follower315-writer-144] [DEBUG] EntryStoreCompleter - follower315: 1 completion(s) pending 11:16:51,936 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: persisted 1 entries in 4.599 ms 11:16:51,936 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryStoreCompleter - follower315: running 1 completion(s) 11:16:51,936 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: received 1 entries in 7.760 μs 11:16:51,936 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] ReplicatedLogImpl - follower315: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 11:16:51,936 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryStoreCompleter - follower315: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c087b3c8@6ae72da5 11:16:51,936 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: persisted 1 entries in 112.0 μs 11:16:51,936 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: waiting for more work [WARN] [06/27/2025 11:16:51.936] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower315] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower315, initialSyncDone=true] 11:16:51,939 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower315: onLeaderTransitioning: LeaderTransitioning [leaderId=leader313] 11:16:51,940 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 11:16:51,940 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower315 starting 11:16:51,940 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower315: onShutDown 11:16:51,940 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower315: terminating thread VirtualThread[#720,follower315-writer-144]/waiting 11:16:51,940 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: received 1 entries in 4.166 ms 11:16:51,940 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: persisted 1 entries in 31.13 μs 11:16:51,940 AM [follower315-writer-144] [DEBUG] JournalWriteTask - follower315: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@62f0c9a9 stopped 11:16:51,941 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower315: journal closed 11:16:51,941 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower315: journal closed 11:16:51,941 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower315: stopped executor with 0 remaining tasks 11:16:51,941 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower315: stopped executor with 0 remaining tasks 11:16:51,941 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower315 ending 11:16:51,941 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 11:16:51,941 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower314-notifier317#-1839601272] 11:16:51,952 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower314-collector318#-1033300209] 11:16:51,952 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower314] 11:16:51,952 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower314: terminating thread VirtualThread[#719,follower314-writer-143]/waiting 11:16:51,953 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: received 1 entries in 602.0 ms 11:16:51,953 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: persisted 1 entries in 21.55 μs 11:16:51,953 AM [follower314-writer-143] [DEBUG] JournalWriteTask - follower314: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@531a4b2 stopped 11:16:51,953 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower314: journal closed 11:16:51,953 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower314: journal closed 11:16:51,953 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: stopped executor with 0 remaining tasks 11:16:51,953 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower314: stopped executor with 0 remaining tasks 11:16:51,953 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower315-notifier319#-1836914988] 11:16:51,953 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower315-collector320#-75365255] 11:16:51,953 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower315] 11:16:51,954 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower316-notifier321#377847436] 11:16:51,954 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower316-collector322#-72515465] 11:16:51,954 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower316] 11:16:51,954 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower316: terminating thread VirtualThread[#721,follower316-writer-145]/waiting 11:16:51,954 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: received 1 entries in 603.6 ms 11:16:51,954 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: persisted 1 entries in 21.08 μs 11:16:51,954 AM [follower316-writer-145] [DEBUG] JournalWriteTask - follower316: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@5dd98b31 stopped 11:16:51,955 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower316: journal closed 11:16:51,955 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower316: journal closed 11:16:51,955 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower316: stopped executor with 0 remaining tasks 11:16:51,955 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower316: stopped executor with 0 remaining tasks 11:16:51,955 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader313-notifier323#-1485215172] 11:16:51,955 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader313-collector324#-1526715569] 11:16:51,955 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader313] 11:16:51,958 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [06/27/2025 11:16:51.959] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader325]: org.opendaylight.controller.cluster.common.actor.Monitor@b9582ee 11:16:51,960 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - leader325: Starting recovery with journal batch size 1000 11:16:51,960 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader325: started executor 11:16:51,960 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader325: started executor 11:16:51,961 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader325: metadata journal open: replayFrom=1 applyTo=0 11:16:51,961 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader325: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:51,961 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader325: journal open: applyTo=0 11:16:51,961 AM [leader325-writer-147] [DEBUG] JournalWriteTask - leader325: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@e1ddfc7 started 11:16:51,961 AM [leader325-writer-147] [DEBUG] JournalWriteTask - leader325: waiting for more work 11:16:51,963 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7046126842443184064/leader325/TermInfo.properties does not exist 11:16:51,963 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader325: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7046126842443184064/leader325/journal-v1-1.log 11:16:51,963 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader325: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest7046126842443184064/leader325/metadata-v1-1.log 11:16:51,963 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader325: no eligible files found 11:16:51,963 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader325: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:51,966 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader325: Set currentTerm=0, votedFor=null 11:16:51,966 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader325: Local TermInfo store seeded with TermInfo{term=0} 11:16:51,966 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader325: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@cc1e679 11:16:51,966 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractReplicatedLog - leader325: Moving last applied index from -1 to -1 11:16:51,966 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] JournalRecovery - leader325: journal recovery completedin 213.7 μs with journalIndex=1 11:16:51,966 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:51,966 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - leader325 (Follower): Received TimeoutNow - switching to Candidate 11:16:51,966 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader325 (Candidate): Election: Candidate has following voting peers: [] 11:16:51,970 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader325: Set currentTerm=1, votedFor=leader325 11:16:51,970 AM [test-pekko.actor.default-dispatcher-18] [INFO] Candidate - leader325 (Candidate): Starting new election term 1 11:16:51,970 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader325 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 11:16:51,970 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader325 (Candidate): Received ElectionTimeout 11:16:51,970 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader325 (Leader): Election: Leader has following peers: [] 11:16:51,970 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader325 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 11:16:51,970 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader325 starting 11:16:51,970 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader325: onShutDown 11:16:51,971 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - leader325: terminating thread VirtualThread[#723,leader325-writer-147]/waiting 11:16:51,971 AM [leader325-writer-147] [DEBUG] JournalWriteTask - leader325: received 1 entries in 9.153 ms 11:16:51,971 AM [leader325-writer-147] [DEBUG] JournalWriteTask - leader325: persisted 1 entries in 22.98 μs 11:16:51,971 AM [leader325-writer-147] [DEBUG] JournalWriteTask - leader325: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@e1ddfc7 stopped 11:16:51,971 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader325: journal closed 11:16:51,971 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader325: journal closed 11:16:51,971 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader325: stopped executor with 0 remaining tasks 11:16:51,971 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader325: stopped executor with 0 remaining tasks 11:16:51,971 AM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader325 ending 11:16:51,971 AM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 11:16:51,971 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader325-collector329#1767439395] 11:16:51,982 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader325] 11:16:51,983 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 11:16:51,983 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/27/2025 11:16:51.985] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower331]: org.opendaylight.controller.cluster.common.actor.Monitor@2fbe2f75 11:16:51,986 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - follower331: Starting recovery with journal batch size 1000 11:16:51,986 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: started executor 11:16:51,986 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: started executor 11:16:51,986 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower331: metadata journal open: replayFrom=1 applyTo=0 11:16:51,987 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower331: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:51,987 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower331: journal open: applyTo=0 11:16:51,987 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@5c64d4ce started 11:16:51,987 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: waiting for more work 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower331/TermInfo.properties does not exist 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower331/journal-v1-1.log 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower331/metadata-v1-1.log 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: no eligible files found 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower331: SnapshotOffer called. 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower331: initializing from snapshot taken at 2025-06-27T11:16:51.983Z 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower331: Set currentTerm=1, votedFor=null 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower331: Updated server config: isVoting: true, peers: [PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=NON_VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING], PeerInfo [id=leader330, address=pekko://test/user/leader330, votingState=VOTING]] 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower331 in 95.80 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower331: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:51.983Z 11:16:51,988 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower331: support() called 11:16:51,989 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower331/snapshot-00000000685E7DA3-3A9763C0.tmp [WARN] [06/27/2025 11:16:51.989] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower332]: org.opendaylight.controller.cluster.common.actor.Monitor@7757ed35 11:16:51,989 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActor - follower332: Starting recovery with journal batch size 1000 11:16:51,989 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: started executor 11:16:51,989 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: started executor 11:16:51,990 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower332: metadata journal open: replayFrom=1 applyTo=0 11:16:51,991 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower332: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:51,991 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower332: journal open: applyTo=0 11:16:51,991 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@53309c3b started 11:16:51,991 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: waiting for more work 11:16:51,991 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower331/snapshot-00000000685E7DA3-3A9763C0.v1 11:16:51,992 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower331/snapshot-00000000685E7DA3-3A9763C0.v1 11:16:51,992 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower331/journal-v1-1.log 11:16:51,992 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower331/metadata-v1-1.log 11:16:51,992 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:51.983Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower331/snapshot-00000000685E7DA3-3A9763C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:51,992 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower332/TermInfo.properties does not exist 11:16:51,992 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower331: local snapshot saved in 3.973 ms, deleting Pekko-persisted snapshots 11:16:51,992 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower332/journal-v1-1.log 11:16:51,992 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower332/metadata-v1-1.log 11:16:51,992 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: no eligible files found 11:16:51,992 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower332: SnapshotOffer called. 11:16:51,992 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - follower332: initializing from snapshot taken at 2025-06-27T11:16:51.983Z 11:16:51,992 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=1, votedFor=null 11:16:51,992 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - follower332: Updated server config: isVoting: true, peers: [PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=NON_VOTING], PeerInfo [id=leader330, address=pekko://test/user/leader330, votingState=VOTING], PeerInfo [id=follower331, address=pekko://test/user/follower331, votingState=VOTING]] 11:16:51,993 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for follower332 in 100.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:51,993 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower332: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:51.983Z 11:16:51,993 AM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - follower332: support() called 11:16:51,993 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower332/snapshot-00000000685E7DA3-3A9763C0.tmp 11:16:51,993 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower331: Recovery completed in 5.353 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [06/27/2025 11:16:51.994] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower333]: org.opendaylight.controller.cluster.common.actor.Monitor@1198bcdf 11:16:51,994 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower333: Starting recovery with journal batch size 1000 11:16:51,994 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: started executor 11:16:51,994 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: started executor 11:16:51,995 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower333: metadata journal open: replayFrom=1 applyTo=0 11:16:51,995 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower333: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:51,995 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - follower333: journal open: applyTo=0 11:16:51,996 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@352d5401 started 11:16:51,996 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: waiting for more work 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower333/TermInfo.properties does not exist 11:16:51,996 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower331: Set currentTerm=1, votedFor=null 11:16:51,996 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower331: Local TermInfo store seeded with TermInfo{term=1} 11:16:51,996 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower331: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@2b9c3eaf 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower333/journal-v1-1.log 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower333/metadata-v1-1.log 11:16:51,996 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower331: Moving last applied index from -1 to -1 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: no eligible files found 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower333: SnapshotOffer called. 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower333: initializing from snapshot taken at 2025-06-27T11:16:51.983Z 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=1, votedFor=null 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower333: Updated server config: isVoting: false, peers: [PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING], PeerInfo [id=leader330, address=pekko://test/user/leader330, votingState=VOTING], PeerInfo [id=follower331, address=pekko://test/user/follower331, votingState=VOTING]] 11:16:51,996 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower331: journal recovery completedin 183.4 μs with journalIndex=1 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for follower333 in 66.85 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower333: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:51.983Z 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower333: support() called 11:16:51,996 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower332/snapshot-00000000685E7DA3-3A9763C0.v1 11:16:51,996 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower332/snapshot-00000000685E7DA3-3A9763C0.v1 11:16:51,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower333/snapshot-00000000685E7DA3-3A9763C0.tmp 11:16:51,996 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower332/journal-v1-1.log 11:16:51,996 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower332/metadata-v1-1.log 11:16:51,996 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower331: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:51,996 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower332: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:51.983Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower332/snapshot-00000000685E7DA3-3A9763C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:51,997 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - follower332: local snapshot saved in 3.847 ms, deleting Pekko-persisted snapshots 11:16:51,997 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower332: Recovery completed in 4.981 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [06/27/2025 11:16:51.998] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader330]: org.opendaylight.controller.cluster.common.actor.Monitor@7b10b9ba 11:16:51,999 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - leader330: Starting recovery with journal batch size 1000 11:16:51,999 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower333/snapshot-00000000685E7DA3-3A9763C0.v1 11:16:51,999 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: started executor 11:16:51,999 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: started executor 11:16:51,999 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower333/snapshot-00000000685E7DA3-3A9763C0.v1 11:16:51,999 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower333/journal-v1-1.log 11:16:51,999 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower333/metadata-v1-1.log 11:16:51,999 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower333: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:51.983Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/follower333/snapshot-00000000685E7DA3-3A9763C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:52,0 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower333: local snapshot saved in 3.412 ms, deleting Pekko-persisted snapshots 11:16:52,0 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=1, votedFor=null 11:16:52,0 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower332: Local TermInfo store seeded with TermInfo{term=1} 11:16:52,0 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower332: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@1090602 11:16:52,0 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower332: Moving last applied index from -1 to -1 11:16:52,0 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower333: Recovery completed in 4.091 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:52,0 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower332: journal recovery completedin 183.5 μs with journalIndex=1 11:16:52,0 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader330: metadata journal open: replayFrom=1 applyTo=0 11:16:52,0 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower332: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:52,1 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader330: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:52,1 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader330: journal open: applyTo=0 11:16:52,2 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@55ae5684 started 11:16:52,2 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: waiting for more work 11:16:52,2 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=1, votedFor=null 11:16:52,2 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower333: Local TermInfo store seeded with TermInfo{term=1} 11:16:52,2 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower333: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@17012147 11:16:52,2 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower333: Moving last applied index from -1 to -1 11:16:52,2 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - follower333: journal recovery completedin 175.8 μs with journalIndex=1 11:16:52,3 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower333: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/leader330/TermInfo.properties does not exist 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/leader330/journal-v1-1.log 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/leader330/metadata-v1-1.log 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: no eligible files found 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - leader330: SnapshotOffer called. 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - leader330: initializing from snapshot taken at 2025-06-27T11:16:51.983Z 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader330: Set currentTerm=1, votedFor=null 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - leader330: Updated server config: isVoting: true, peers: [PeerInfo [id=follower333, address=pekko://test/user/follower333, votingState=NON_VOTING], PeerInfo [id=follower332, address=pekko://test/user/follower332, votingState=VOTING], PeerInfo [id=follower331, address=pekko://test/user/follower331, votingState=VOTING]] 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for leader330 in 54.31 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader330: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:51.983Z 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - leader330: support() called 11:16:52,3 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/leader330/snapshot-00000000685E7DA3-3A9763C0.tmp 11:16:52,5 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/leader330/snapshot-00000000685E7DA3-3A9763C0.v1 11:16:52,5 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/leader330/snapshot-00000000685E7DA3-3A9763C0.v1 11:16:52,5 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/leader330/journal-v1-1.log 11:16:52,5 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/leader330/metadata-v1-1.log 11:16:52,5 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:51.983Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest6457404817186730297/leader330/snapshot-00000000685E7DA3-3A9763C0.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:52,5 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader330: local snapshot saved in 2.336 ms, deleting Pekko-persisted snapshots 11:16:52,6 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader330: Recovery completed in 3.118 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:52,8 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - leader330: Set currentTerm=1, votedFor=null 11:16:52,8 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - leader330: Local TermInfo store seeded with TermInfo{term=1} 11:16:52,8 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader330: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@1adc78ee 11:16:52,8 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractReplicatedLog - leader330: Moving last applied index from -1 to -1 11:16:52,8 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] JournalRecovery - leader330: journal recovery completedin 191.1 μs with journalIndex=1 11:16:52,9 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader330: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:52,360 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:52,361 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader330 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.168:17355, Removed) 11:16:52,361 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - leader330 (Follower): Received ElectionTimeout - switching to Candidate 11:16:52,361 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader330 (Candidate): Election: Candidate has following voting peers: [follower332, follower331] 11:16:52,364 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader330: Set currentTerm=2, votedFor=leader330 11:16:52,364 AM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - leader330 (Candidate): Starting new election term 2 11:16:52,364 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader330 (Candidate): Sending RequestVote{term=2, candidateId=leader330, lastLogIndex=-1, lastLogTerm=-1} to peer follower332 11:16:52,364 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - leader330 (Candidate): Sending RequestVote{term=2, candidateId=leader330, lastLogIndex=-1, lastLogTerm=-1} to peer follower331 11:16:52,364 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader330 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:16:52,365 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower332 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:52,365 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower332 (Follower): Cluster state: [] 11:16:52,365 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower332 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader330, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:52,365 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower332 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader330, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:52,365 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower331 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:52,365 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower331 (Follower): Cluster state: [] 11:16:52,365 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower331 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader330, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:52,365 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower331 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader330, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:52,367 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=2, votedFor=null 11:16:52,367 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower331: Set currentTerm=2, votedFor=null 11:16:52,367 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower332 (Follower): In requestVote: RequestVote{term=2, candidateId=leader330, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:52,367 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower331 (Follower): In requestVote: RequestVote{term=2, candidateId=leader330, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:52,370 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower331: Set currentTerm=2, votedFor=leader330 11:16:52,370 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower331 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:52,371 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader330 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 11:16:52,371 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader330 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 11:16:52,371 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader330 (Leader): Election: Leader has following peers: [follower333, follower332, follower331] 11:16:52,371 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower333: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:52,371 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower332: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:52,371 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower331: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:52,371 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower331: Last sync leader does not match current leader leader330, need to catch up to -1 11:16:52,371 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader330 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 [WARN] [06/27/2025 11:16:52.371] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/follower331] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower331, initialSyncDone=false] 11:16:52,372 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader330 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower331 updated: matchIndex: -1, nextIndex: 0 11:16:52,372 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower331: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:52,371 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower333 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader330, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 11:16:52,372 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower332: Set currentTerm=2, votedFor=leader330 11:16:52,372 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower332 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:52,372 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader330 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower332 updated: matchIndex: -1, nextIndex: 0 11:16:52,372 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:52,372 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower332: Last sync leader does not match current leader leader330, need to catch up to -1 [WARN] [06/27/2025 11:16:52.372] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower332] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower332, initialSyncDone=false] 11:16:52,374 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower333: Set currentTerm=2, votedFor=null 11:16:52,374 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower333: Last sync leader does not match current leader leader330, need to catch up to -1 11:16:52,374 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower333 updated: matchIndex: -1, nextIndex: 0 [WARN] [06/27/2025 11:16:52.375] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower333] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower333, initialSyncDone=false] 11:16:52,375 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:52,415 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:16:52,415 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:16:52,416 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:16:52,416 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 11:16:52,416 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader330: Persist data index=0 term=2 command=zero 11:16:52,416 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader330: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 11:16:52,416 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 11:16:52,416 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:52,416 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): sendAppendEntries: 0 is present for follower follower333 11:16:52,416 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: received 1 entries in 414.7 ms 11:16:52,416 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): Sending AppendEntries to follower follower333: AppendEntries{term=2, leaderId=leader330, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 11:16:52,416 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower333 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader330, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 11:16:52,416 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower333 (Follower): Number of entries to be appended = 1 11:16:52,416 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower333 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:16:52,416 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower333 (Follower): Append entry to log zero 11:16:52,417 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower333: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 11:16:52,417 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower333 (Follower): Log size is now 1 11:16:52,417 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower333 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower333, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 11:16:52,417 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower333: Lagging 0 entries behind leader leader330 and reached -1 (of expected -1) 11:16:52,417 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryStoreCompleter - follower333: awaiting more completions to resolve 1 deferred callback(s) 11:16:52,417 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: received 1 entries in 421.1 ms 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): sendAppendEntries: 0 is present for follower follower332 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): Sending AppendEntries to follower follower332: AppendEntries{term=2, leaderId=leader330, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower331: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): sendAppendEntries: 0 is present for follower follower331 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): Sending AppendEntries to follower follower331: AppendEntries{term=2, leaderId=leader330, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower333 updated: matchIndex: 0, nextIndex: 1 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower331 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader330, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=true}]} 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower331 (Follower): Number of entries to be appended = 1 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower331 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower331 (Follower): Append entry to log zero 11:16:52,417 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower331: Append log entry and persist JournaledLogEntry{index=0, term=2, command=zero, pending=true} 11:16:52,417 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: received 1 entries in 430.4 ms 11:16:52,418 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower331 (Follower): Log size is now 1 11:16:52,418 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower331 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower331, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 11:16:52,418 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower331: Lagging 0 entries behind leader leader330 and reached -1 (of expected -1) 11:16:52,418 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader330 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower331 updated: matchIndex: 0, nextIndex: 1 11:16:52,418 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower331: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:52,418 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryStoreCompleter - follower331: awaiting more completions to resolve 1 deferred callback(s) 11:16:52,422 AM [leader330-writer-151] [DEBUG] EntryStoreCompleter - leader330: 1 completion(s) pending 11:16:52,422 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryStoreCompleter - leader330: running 1 completion(s) 11:16:52,422 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: persisted 1 entries in 5.458 ms 11:16:52,422 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: waiting for more work 11:16:52,422 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] ReplicatedLogImpl - leader330: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=true} 11:16:52,422 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader330 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 11:16:52,422 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - leader330 (Leader): Setting last applied to 0 11:16:52,422 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - leader330: Moving last applied index from -1 to 0 11:16:52,422 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader330: Applying state for log index 0 data zero 11:16:52,422 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - leader330: applyState called: zero 11:16:52,422 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: received 1 entries in 427.4 μs 11:16:52,422 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: persisted 1 entries in 100.1 μs 11:16:52,422 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: waiting for more work 11:16:52,423 AM [follower331-writer-148] [DEBUG] EntryStoreCompleter - follower331: 1 completion(s) pending 11:16:52,423 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryStoreCompleter - follower331: running 1 completion(s) 11:16:52,423 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: persisted 1 entries in 5.715 ms 11:16:52,423 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: waiting for more work 11:16:52,423 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower331: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 11:16:52,423 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryStoreCompleter - follower331: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c087b3c8@297a2340 11:16:52,423 AM [follower333-writer-150] [DEBUG] EntryStoreCompleter - follower333: 1 completion(s) pending 11:16:52,423 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryStoreCompleter - follower333: running 1 completion(s) 11:16:52,423 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: persisted 1 entries in 6.528 ms 11:16:52,423 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: waiting for more work 11:16:52,423 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower333: persist complete JournaledLogEntry{index=0, term=2, command=zero, pending=false} 11:16:52,423 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryStoreCompleter - follower333: completed deferred callback org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda/0x00000007c087b3c8@41f1e030 [WARN] [06/27/2025 11:16:52.423] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower333] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower333, initialSyncDone=true] [WARN] [06/27/2025 11:16:52.424] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower331] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower331, initialSyncDone=true] 11:16:52,490 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader330 (Leader): sendAppendEntries: 0 is present for follower follower332 11:16:52,490 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader330 (Leader): Sending AppendEntries to follower follower332: AppendEntries{term=2, leaderId=leader330, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 11:16:52,610 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader330 (Leader): sendAppendEntries: 0 is present for follower follower332 11:16:52,610 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader330 (Leader): Sending AppendEntries to follower follower332: AppendEntries{term=2, leaderId=leader330, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=2, command=zero, pending=false}]} 11:16:52,610 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower333 (Follower): Commit index set to 0 11:16:52,610 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower333: Lagging 0 entries behind leader leader330 and reached 0 (of expected -1) 11:16:52,610 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - follower333 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:16:52,610 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorBehavior - follower333 (Follower): Setting last applied to 0 11:16:52,610 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower333: Moving last applied index from -1 to 0 11:16:52,610 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower333: Applying state for log index 0 data zero 11:16:52,610 AM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - follower333: applyState called: zero 11:16:52,611 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower331 (Follower): Commit index set to 0 11:16:52,611 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower331: Lagging 0 entries behind leader leader330 and reached 0 (of expected -1) 11:16:52,611 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower331 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 11:16:52,611 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower331 (Follower): Setting last applied to 0 11:16:52,611 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower331: Moving last applied index from -1 to 0 11:16:52,611 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower331: Applying state for log index 0 data zero 11:16:52,611 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower331: applyState called: zero 11:16:52,611 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: received 1 entries in 187.3 ms 11:16:52,611 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: persisted 1 entries in 153.0 μs 11:16:52,611 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: waiting for more work 11:16:52,611 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: received 1 entries in 187.7 ms 11:16:52,611 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: persisted 1 entries in 132.9 μs 11:16:52,611 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: waiting for more work 11:16:52,617 AM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 11:16:52,617 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:16:52,617 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:16:52,617 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader330: onRequestLeadership RequestLeadership [requestedFollowerId=follower332, replyTo=Actor[pekko://test/user/$c#749600896]] 11:16:52,617 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - leader330: Initiating leader transfer 11:16:52,617 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader330: pauseLeader successfully completed - doing transfer 11:16:52,617 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] Leader - leader330 (Leader): Attempting to transfer leadership 11:16:52,617 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower332: onLeaderTransitioning: LeaderTransitioning [leaderId=leader330] 11:16:52,618 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower333: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:52,618 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower332: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:52,618 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader330 (Leader): sendAppendEntries: 0 is present for follower follower332 11:16:52,618 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - leader330 (Leader): Checking sendAppendEntries for follower follower331: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 11:16:52,618 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower333: onLeaderTransitioning: LeaderTransitioning [leaderId=leader330] 11:16:52,618 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower333: Lagging 0 entries behind leader leader330 and reached 0 (of expected -1) 11:16:52,618 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower331: Lagging 0 entries behind leader leader330 and reached 0 (of expected -1) 11:16:52,618 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - follower331: onLeaderTransitioning: LeaderTransitioning [leaderId=leader330] 11:16:52,730 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower331: Lagging 0 entries behind leader leader330 and reached 0 (of expected -1) 11:16:52,730 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower333: Lagging 0 entries behind leader leader330 and reached 0 (of expected -1) 11:16:52,850 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower331: Lagging 0 entries behind leader leader330 and reached 0 (of expected -1) 11:16:52,850 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower333: Lagging 0 entries behind leader leader330 and reached 0 (of expected -1) 11:16:52,970 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader330: leader transfer aborted 11:16:52,970 AM [test-pekko.actor.default-dispatcher-17] [WARN] RaftActorLeadershipTransferCohort - leader330: Failed to transfer leadership in 352.6 ms 11:16:52,970 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActor - leader330: LeadershipTransfer request from follower332 failed 11:16:52,970 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Leader - leader330 (Leader): Leadership transfer expired 11:16:52,970 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower333: Lagging 0 entries behind leader leader330 and reached 0 (of expected -1) 11:16:52,970 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower331: Lagging 0 entries behind leader leader330 and reached 0 (of expected -1) 11:16:52,971 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 11:16:52,971 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower331-notifier334#271304392] 11:16:52,982 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower331-collector335#-1828729842] 11:16:52,982 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower331] 11:16:52,982 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower331: terminating thread VirtualThread[#724,follower331-writer-148]/waiting 11:16:52,983 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: received 1 entries in 371.5 ms 11:16:52,983 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: persisted 1 entries in 34.06 μs 11:16:52,983 AM [follower331-writer-148] [DEBUG] JournalWriteTask - follower331: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@5c64d4ce stopped 11:16:52,983 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower331: journal closed 11:16:52,983 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower331: journal closed 11:16:52,983 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: stopped executor with 0 remaining tasks 11:16:52,983 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower331: stopped executor with 0 remaining tasks 11:16:52,983 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower332-notifier336#-343607665] 11:16:52,983 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower332-collector337#-276865289] 11:16:52,984 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower332] 11:16:52,984 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower332: terminating thread VirtualThread[#725,follower332-writer-149]/waiting 11:16:52,984 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: received 1 entries in 992.9 ms 11:16:52,984 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: persisted 1 entries in 21.12 μs 11:16:52,984 AM [follower332-writer-149] [DEBUG] JournalWriteTask - follower332: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@53309c3b stopped 11:16:52,984 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower332: journal closed 11:16:52,984 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower332: journal closed 11:16:52,984 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: stopped executor with 0 remaining tasks 11:16:52,984 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower332: stopped executor with 0 remaining tasks 11:16:52,985 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower333-notifier338#1316577019] 11:16:52,985 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower333-collector339#-147420940] 11:16:52,985 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower333] 11:16:52,985 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - follower333: terminating thread VirtualThread[#726,follower333-writer-150]/waiting 11:16:52,985 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: received 1 entries in 374.5 ms 11:16:52,986 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: persisted 1 entries in 21.46 μs 11:16:52,986 AM [follower333-writer-150] [DEBUG] JournalWriteTask - follower333: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@352d5401 stopped 11:16:52,986 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower333: journal closed 11:16:52,986 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower333: journal closed 11:16:52,986 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: stopped executor with 0 remaining tasks 11:16:52,986 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower333: stopped executor with 0 remaining tasks 11:16:52,986 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader330-notifier340#989002628] 11:16:52,986 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader330-collector341#-434510775] 11:16:52,988 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader330] 11:16:52,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRaftStorage - leader330: terminating thread VirtualThread[#727,leader330-writer-151]/waiting 11:16:52,988 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: received 1 entries in 565.5 ms 11:16:52,988 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: persisted 1 entries in 21.03 μs 11:16:52,988 AM [leader330-writer-151] [DEBUG] JournalWriteTask - leader330: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@55ae5684 stopped 11:16:52,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - leader330: journal closed 11:16:52,988 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - leader330: journal closed 11:16:52,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: stopped executor with 0 remaining tasks 11:16:52,988 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - leader330: stopped executor with 0 remaining tasks 11:16:52,990 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$c#749600896] 11:16:52,992 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 11:16:52,992 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [06/27/2025 11:16:52.994] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower343]: org.opendaylight.controller.cluster.common.actor.Monitor@4d5ef336 11:16:52,994 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - follower343: Starting recovery with journal batch size 1000 11:16:52,994 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower343: started executor 11:16:52,994 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower343: started executor 11:16:52,995 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower343: metadata journal open: replayFrom=1 applyTo=0 11:16:52,995 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower343: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:52,995 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower343: journal open: applyTo=0 11:16:52,995 AM [follower343-writer-152] [DEBUG] JournalWriteTask - follower343: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@327295fa started 11:16:52,995 AM [follower343-writer-152] [DEBUG] JournalWriteTask - follower343: waiting for more work 11:16:52,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower343/TermInfo.properties does not exist 11:16:52,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower343/journal-v1-1.log 11:16:52,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower343/metadata-v1-1.log 11:16:52,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower343: no eligible files found 11:16:52,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower343: SnapshotOffer called. 11:16:52,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PekkoRecovery - follower343: initializing from snapshot taken at 2025-06-27T11:16:52.992Z 11:16:52,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower343: Set currentTerm=1, votedFor=null 11:16:52,996 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower343: Updated server config: isVoting: true, peers: [PeerInfo [id=leader342, address=pekko://test/user/leader342, votingState=VOTING], PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=NON_VOTING]] 11:16:52,997 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - Recovery snapshot applied for follower343 in 100.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:52,997 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower343: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:52.992Z 11:16:52,997 AM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - follower343: support() called 11:16:52,997 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower343: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower343/snapshot-00000000685E7DA4-3B20B800.tmp [WARN] [06/27/2025 11:16:52.998] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower344]: org.opendaylight.controller.cluster.common.actor.Monitor@347900fb 11:16:52,998 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActor - follower344: Starting recovery with journal batch size 1000 11:16:52,998 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: started executor 11:16:52,998 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: started executor 11:16:52,999 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower344: metadata journal open: replayFrom=1 applyTo=0 11:16:53,0 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] EntryJournalV1 - follower344: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:53,0 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRaftStorage - follower344: journal open: applyTo=0 11:16:53,0 AM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@88f6182 started 11:16:53,0 AM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: waiting for more work 11:16:53,0 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower344/TermInfo.properties does not exist 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower344/journal-v1-1.log 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower344/metadata-v1-1.log 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: no eligible files found 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower344: SnapshotOffer called. 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PekkoRecovery - follower344: initializing from snapshot taken at 2025-06-27T11:16:52.992Z 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=1, votedFor=null 11:16:53,1 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower343: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower343/snapshot-00000000685E7DA4-3B20B800.v1 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PeerInfos - follower344: Updated server config: isVoting: true, peers: [PeerInfo [id=leader342, address=pekko://test/user/leader342, votingState=VOTING], PeerInfo [id=follower343, address=pekko://test/user/follower343, votingState=VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=NON_VOTING]] 11:16:53,1 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower343: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower343/snapshot-00000000685E7DA4-3B20B800.v1 11:16:53,1 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower343/journal-v1-1.log 11:16:53,1 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower343: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower343/metadata-v1-1.log 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - Recovery snapshot applied for follower344 in 75.03 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower344: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:52.992Z 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [INFO] MockRaftActor - follower344: support() called 11:16:53,1 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower343: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:52.992Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower343/snapshot-00000000685E7DA4-3B20B800.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:53,1 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRecovery - follower343: local snapshot saved in 4.341 ms, deleting Pekko-persisted snapshots 11:16:53,1 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower344/snapshot-00000000685E7DA4-3B20B800.tmp 11:16:53,2 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower343: Recovery completed in 5.272 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 [WARN] [06/27/2025 11:16:53.002] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower345]: org.opendaylight.controller.cluster.common.actor.Monitor@274afdaf 11:16:53,2 AM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - follower345: Starting recovery with journal batch size 1000 11:16:53,2 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: started executor 11:16:53,3 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: started executor 11:16:53,3 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower345: metadata journal open: replayFrom=1 applyTo=0 11:16:53,3 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower345: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:53,3 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRaftStorage - follower345: journal open: applyTo=0 11:16:53,3 AM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3bbc2eef started 11:16:53,3 AM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: waiting for more work 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower345/TermInfo.properties does not exist 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower345/journal-v1-1.log 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower345/metadata-v1-1.log 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: no eligible files found 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower345: SnapshotOffer called. 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PekkoRecovery - follower345: initializing from snapshot taken at 2025-06-27T11:16:52.992Z 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=1, votedFor=null 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - follower345: Updated server config: isVoting: false, peers: [PeerInfo [id=leader342, address=pekko://test/user/leader342, votingState=VOTING], PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower343, address=pekko://test/user/follower343, votingState=VOTING]] 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - Recovery snapshot applied for follower345 in 82.69 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower345: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:52.992Z 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - follower345: support() called 11:16:53,4 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower345/snapshot-00000000685E7DA4-3B20B800.tmp [WARN] [06/27/2025 11:16:53.005] [test-pekko.actor.default-dispatcher-7] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader342]: org.opendaylight.controller.cluster.common.actor.Monitor@393c1f97 11:16:53,5 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader342: Starting recovery with journal batch size 1000 11:16:53,6 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader342: started executor 11:16:53,6 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader342: started executor 11:16:53,6 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader342: metadata journal open: replayFrom=1 applyTo=0 11:16:53,7 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower344/snapshot-00000000685E7DA4-3B20B800.v1 11:16:53,8 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - follower343: Set currentTerm=1, votedFor=null 11:16:53,8 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRecovery - follower343: Local TermInfo store seeded with TermInfo{term=1} 11:16:53,8 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower343: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@3f57defa 11:16:53,8 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - follower343: Moving last applied index from -1 to -1 11:16:53,8 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower344/snapshot-00000000685E7DA4-3B20B800.v1 11:16:53,8 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] JournalRecovery - follower343: journal recovery completedin 194.0 μs with journalIndex=1 11:16:53,8 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower344/journal-v1-1.log 11:16:53,8 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower344/metadata-v1-1.log 11:16:53,8 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftStorage - follower344: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:52.992Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower344/snapshot-00000000685E7DA4-3B20B800.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:53,8 AM [test-pekko.actor.default-dispatcher-17] [INFO] PekkoRecovery - follower344: local snapshot saved in 7.199 ms, deleting Pekko-persisted snapshots 11:16:53,8 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - follower343: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:53,8 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower345/snapshot-00000000685E7DA4-3B20B800.v1 11:16:53,8 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower345/snapshot-00000000685E7DA4-3B20B800.v1 11:16:53,8 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower345/journal-v1-1.log 11:16:53,9 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower345/metadata-v1-1.log 11:16:53,9 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower345: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:52.992Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/follower345/snapshot-00000000685E7DA4-3B20B800.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:53,9 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower345: local snapshot saved in 4.516 ms, deleting Pekko-persisted snapshots 11:16:53,9 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower344: Recovery completed in 8.105 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:53,9 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader342: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 11:16:53,9 AM [test-pekko.actor.default-dispatcher-12] [INFO] PekkoRaftStorage - leader342: journal open: applyTo=0 11:16:53,9 AM [leader342-writer-155] [DEBUG] JournalWriteTask - leader342: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@6986601c started 11:16:53,9 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower345: Recovery completed in 5.274 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:53,9 AM [leader342-writer-155] [DEBUG] JournalWriteTask - leader342: waiting for more work 11:16:53,10 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/leader342/TermInfo.properties does not exist 11:16:53,10 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader342: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/leader342/journal-v1-1.log 11:16:53,10 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader342: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/leader342/metadata-v1-1.log 11:16:53,10 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader342: no eligible files found 11:16:53,10 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - leader342: SnapshotOffer called. 11:16:53,10 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRecovery - leader342: initializing from snapshot taken at 2025-06-27T11:16:52.992Z 11:16:53,10 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader342: Set currentTerm=1, votedFor=null 11:16:53,10 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PeerInfos - leader342: Updated server config: isVoting: true, peers: [PeerInfo [id=follower344, address=pekko://test/user/follower344, votingState=VOTING], PeerInfo [id=follower343, address=pekko://test/user/follower343, votingState=VOTING], PeerInfo [id=follower345, address=pekko://test/user/follower345, votingState=NON_VOTING]] 11:16:53,10 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - Recovery snapshot applied for leader342 in 122.8 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 11:16:53,11 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader342: migrating from Pekko persistent-snapshot taken at 2025-06-27T11:16:52.992Z 11:16:53,11 AM [test-pekko.actor.default-dispatcher-18] [INFO] MockRaftActor - leader342: support() called 11:16:53,11 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader342: starting snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/leader342/snapshot-00000000685E7DA4-3B20B800.tmp 11:16:53,11 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=1, votedFor=null 11:16:53,11 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=1, votedFor=null 11:16:53,11 AM [test-pekko.actor.default-dispatcher-5] [INFO] PekkoRecovery - follower344: Local TermInfo store seeded with TermInfo{term=1} 11:16:53,11 AM [test-pekko.actor.default-dispatcher-7] [INFO] PekkoRecovery - follower345: Local TermInfo store seeded with TermInfo{term=1} 11:16:53,11 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower344: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@2dc56f67 11:16:53,11 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower345: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@de0629 11:16:53,11 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower344: Moving last applied index from -1 to -1 11:16:53,11 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] AbstractReplicatedLog - follower345: Moving last applied index from -1 to -1 11:16:53,11 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] JournalRecovery - follower345: journal recovery completedin 204.3 μs with journalIndex=1 11:16:53,11 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - follower344: journal recovery completedin 208.4 μs with journalIndex=1 11:16:53,12 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] RaftActor - follower345: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:53,12 AM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower344: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:53,14 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader342: finished snapshot writeout to /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/leader342/snapshot-00000000685E7DA4-3B20B800.v1 11:16:53,14 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader342: selected SNAPSHOT_V1 to handle file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/leader342/snapshot-00000000685E7DA4-3B20B800.v1 11:16:53,14 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader342: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/leader342/journal-v1-1.log 11:16:53,14 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader342: skipping unrecognized file /tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/leader342/metadata-v1-1.log 11:16:53,14 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - leader342: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-06-27T11:16:52.992Z, file=/tmp/org.opendaylight.controller.cluster.raft.LeadershipTransferIntegrationTest8964676860064040024/leader342/snapshot-00000000685E7DA4-3B20B800.v1, size=275, entryCompress=NONE, stateCompress=NONE} 11:16:53,14 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader342: local snapshot saved in 3.370 ms, deleting Pekko-persisted snapshots 11:16:53,15 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader342: Recovery completed in 4.254 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 11:16:53,17 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader342: Set currentTerm=1, votedFor=null 11:16:53,17 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRecovery - leader342: Local TermInfo store seeded with TermInfo{term=1} 11:16:53,17 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader342: Pekko recovery completed with org.opendaylight.controller.cluster.raft.RecoveryLog@3995e61b 11:16:53,17 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractReplicatedLog - leader342: Moving last applied index from -1 to -1 11:16:53,17 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] JournalRecovery - leader342: journal recovery completedin 238.2 μs with journalIndex=1 11:16:53,18 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader342: ignoring DeleteSnapshotsSuccess(SnapshotSelectionCriteria(9223372036854775807,9223372036854775807,0,0)) 11:16:53,410 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:53,410 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - leader342 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.168:17355, Removed) 11:16:53,410 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Follower - leader342 (Follower): Received ElectionTimeout - switching to Candidate 11:16:53,410 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader342 (Candidate): Election: Candidate has following voting peers: [follower344, follower343] 11:16:53,414 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - leader342: Set currentTerm=2, votedFor=leader342 11:16:53,414 AM [test-pekko.actor.default-dispatcher-18] [INFO] Candidate - leader342 (Candidate): Starting new election term 2 11:16:53,414 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader342 (Candidate): Sending RequestVote{term=2, candidateId=leader342, lastLogIndex=-1, lastLogTerm=-1} to peer follower344 11:16:53,414 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader342 (Candidate): Sending RequestVote{term=2, candidateId=leader342, lastLogIndex=-1, lastLogTerm=-1} to peer follower343 11:16:53,414 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader342 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 11:16:53,414 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower344 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:53,414 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower344 (Follower): Cluster state: [] 11:16:53,414 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower344 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader342, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:53,414 AM [test-pekko.actor.default-dispatcher-18] [INFO] Follower - follower344 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader342, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:53,414 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower343 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:53,415 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower343 (Follower): Cluster state: [] 11:16:53,415 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower343 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader342, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:53,415 AM [test-pekko.actor.default-dispatcher-17] [INFO] Follower - follower343 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader342, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 11:16:53,418 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=2, votedFor=null 11:16:53,418 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower344 (Follower): In requestVote: RequestVote{term=2, candidateId=leader342, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:53,419 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower343: Set currentTerm=2, votedFor=null 11:16:53,419 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower343 (Follower): In requestVote: RequestVote{term=2, candidateId=leader342, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:53,421 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=2, votedFor=leader342 11:16:53,421 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorBehavior - follower344 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:53,421 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader342 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 11:16:53,421 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Candidate - leader342 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 11:16:53,422 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Election: Leader has following peers: [follower344, follower343, follower345] 11:16:53,422 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower344: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,422 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower343: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,422 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower345: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,422 AM [test-pekko.actor.default-dispatcher-7] [DEBUG] SyncStatusTracker - follower344: Last sync leader does not match current leader leader342, need to catch up to -1 11:16:53,422 AM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - follower345 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader342, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 11:16:53,422 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] PropertiesTermInfoStore - follower343: Set currentTerm=2, votedFor=leader342 11:16:53,422 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorBehavior - follower343 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 11:16:53,422 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower343: Last sync leader does not match current leader leader342, need to catch up to -1 [WARN] [06/27/2025 11:16:53.422] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower344] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower344, initialSyncDone=false] [WARN] [06/27/2025 11:16:53.422] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower343] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower343, initialSyncDone=false] 11:16:53,422 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorBehavior - leader342 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 11:16:53,423 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower344 updated: matchIndex: -1, nextIndex: 0 11:16:53,423 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,423 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower343 updated: matchIndex: -1, nextIndex: 0 11:16:53,423 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower343: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,424 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower345: Set currentTerm=2, votedFor=null 11:16:53,424 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower345: Last sync leader does not match current leader leader342, need to catch up to -1 11:16:53,424 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader342 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower345 updated: matchIndex: -1, nextIndex: 0 11:16:53,424 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:53.425] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower345] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower345, initialSyncDone=false] 11:16:53,471 AM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 11:16:53,471 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 11:16:53,471 AM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 11:16:53,471 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower343] 11:16:53,482 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower343: terminating thread VirtualThread[#728,follower343-writer-152]/waiting 11:16:53,482 AM [follower343-writer-152] [DEBUG] JournalWriteTask - follower343: received 1 entries in 487.0 ms 11:16:53,483 AM [follower343-writer-152] [DEBUG] JournalWriteTask - follower343: persisted 1 entries in 33.68 μs 11:16:53,483 AM [follower343-writer-152] [DEBUG] JournalWriteTask - follower343: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@327295fa stopped 11:16:53,483 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower343: journal closed 11:16:53,483 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower343: journal closed 11:16:53,483 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower343: stopped executor with 0 remaining tasks 11:16:53,483 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower343: stopped executor with 0 remaining tasks 11:16:53,483 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower345] 11:16:53,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] PekkoRaftStorage - follower345: terminating thread VirtualThread[#730,follower345-writer-154]/waiting 11:16:53,494 AM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: received 1 entries in 490.8 ms 11:16:53,494 AM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: persisted 1 entries in 21.82 μs 11:16:53,494 AM [follower345-writer-154] [DEBUG] JournalWriteTask - follower345: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@3bbc2eef stopped 11:16:53,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] EntryJournalV1 - follower345: journal closed 11:16:53,494 AM [test-pekko.actor.default-dispatcher-18] [INFO] PekkoRaftStorage - follower345: journal closed 11:16:53,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower345: stopped executor with 0 remaining tasks 11:16:53,494 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftStorage - follower345: stopped executor with 0 remaining tasks 11:16:53,495 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 11:16:53,495 AM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 11:16:53,495 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader342: onRequestLeadership RequestLeadership [requestedFollowerId=follower344, replyTo=Actor[pekko://test/user/$d#-1100830818]] 11:16:53,495 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader342: Initiating leader transfer 11:16:53,495 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower344: onLeaderTransitioning: LeaderTransitioning [leaderId=leader342] [WARN] [06/27/2025 11:16:53.495] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower343] received dead letter from TestActor[pekko://test/user/leader342]: LeaderTransitioning [leaderId=leader342] 11:16:53,495 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader342: pauseLeader successfully completed - doing transfer 11:16:53,495 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Leader - leader342 (Leader): Attempting to transfer leadership [WARN] [06/27/2025 11:16:53.495] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower345] received dead letter from TestActor[pekko://test/user/leader342]: LeaderTransitioning [leaderId=leader342] 11:16:53,495 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,495 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower343: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,496 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower344: Lagging 0 entries behind leader leader342 and reached -1 (of expected -1) [WARN] [06/27/2025 11:16:53.496] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower343] received dead letter from TestActor[pekko://test/user/leader342]: AppendEntries{term=2, leaderId=leader342, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [06/27/2025 11:16:53.496] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower344] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower344, initialSyncDone=true] 11:16:53,496 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:53.496] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower345] received dead letter from TestActor[pekko://test/user/leader342]: AppendEntries{term=2, leaderId=leader342, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 11:16:53,496 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Leader - leader342 (Leader): tryToCompleteLeadershipTransfer: followerId: follower344, matchIndex: -1, lastIndex: -1, isVoting: true 11:16:53,496 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Leader - leader342 (Leader): Follower's log matches - sending ElectionTimeout 11:16:53,496 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower344: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,496 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower344: Lagging 0 entries behind leader leader342 and reached -1 (of expected -1) 11:16:53,496 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower343: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:53.496] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower343] received dead letter from TestActor[pekko://test/user/leader342]: AppendEntries{term=2, leaderId=leader342, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 11:16:53,497 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractLeader - leader342 (Leader): Checking sendAppendEntries for follower follower345: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:53.497] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower345] received dead letter from TestActor[pekko://test/user/leader342]: AppendEntries{term=2, leaderId=leader342, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 11:16:53,497 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 11:16:53,497 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower344 (Follower): Received TimeoutNow - switching to Candidate 11:16:53,497 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - follower344 (Candidate): Election: Candidate has following voting peers: [leader342, follower343] 11:16:53,498 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] Leader - leader342 (Leader): Leader transfer complete 11:16:53,498 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader342: leader transfer complete - waiting for new leader 11:16:53,500 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower344: Set currentTerm=3, votedFor=follower344 11:16:53,500 AM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - follower344 (Candidate): Starting new election term 3 11:16:53,500 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - follower344 (Candidate): Sending RequestVote{term=3, candidateId=follower344, lastLogIndex=-1, lastLogTerm=-1} to peer leader342 11:16:53,500 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - follower344 (Candidate): Sending RequestVote{term=3, candidateId=follower344, lastLogIndex=-1, lastLogTerm=-1} to peer follower343 11:16:53,500 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - follower344 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 11:16:53,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader342 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 11:16:53,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader342 (Leader): Cluster state: [] 11:16:53,501 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader342 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower344, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 11:16:53,501 AM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractLeader - leader342 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower344, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower [WARN] [06/27/2025 11:16:53.501] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower343] received dead letter from TestActor[pekko://test/user/follower344]: RequestVote{term=3, candidateId=follower344, lastLogIndex=-1, lastLogTerm=-1} 11:16:53,503 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader342: Set currentTerm=3, votedFor=null 11:16:53,503 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader342 (Leader): Leadership transfer in progress - processing RequestVote 11:16:53,503 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader342 (Leader): In requestVote: RequestVote{term=3, candidateId=follower344, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 11:16:53,506 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader342: Set currentTerm=3, votedFor=follower344 11:16:53,506 AM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader342 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 11:16:53,506 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - follower344 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 11:16:53,506 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] Candidate - follower344 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 11:16:53,506 AM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - leader342 (Leader) :- Switching from behavior Leader to Follower, election term: 3 11:16:53,506 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower344 (Leader): Election: Leader has following peers: [leader342, follower343, follower345] 11:16:53,507 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower344 (Leader): Checking sendAppendEntries for follower leader342: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,507 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower344 (Leader): Checking sendAppendEntries for follower follower343: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,507 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - leader342: Last sync leader does not match current leader follower344, need to catch up to -1 11:16:53,507 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower344 (Leader): Checking sendAppendEntries for follower follower345: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 11:16:53,507 AM [test-pekko.actor.default-dispatcher-17] [INFO] RaftActorBehavior - follower344 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [06/27/2025 11:16:53.507] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower343] received dead letter from TestActor[pekko://test/user/follower344]: AppendEntries{term=3, leaderId=follower344, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} [WARN] [06/27/2025 11:16:53.507] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower345] received dead letter from TestActor[pekko://test/user/follower344]: AppendEntries{term=3, leaderId=follower344, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 11:16:53,507 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader342: leader changed to follower344 11:16:53,507 AM [test-pekko.actor.default-dispatcher-18] [INFO] RaftActorLeadershipTransferCohort - leader342: Successfully transferred leadership to follower344 in 12.12 ms 11:16:53,507 AM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActor - leader342: Leadership transferred successfully to follower344 11:16:53,507 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower344 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader342 updated: matchIndex: -1, nextIndex: 0 11:16:53,507 AM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractLeader - follower344 (Leader): Checking sendAppendEntries for follower leader342: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [06/27/2025 11:16:53.507] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader342] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader342, initialSyncDone=false] 11:16:53,546 AM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 11:16:53,546 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower343-notifier346#1958648389] 11:16:53,557 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower343-collector347#-1020250593] 11:16:53,557 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower344-notifier348#-207914146] 11:16:53,557 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower344-collector349#130144249] 11:16:53,558 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower344] 11:16:53,558 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - follower344: terminating thread VirtualThread[#729,follower344-writer-153]/waiting 11:16:53,558 AM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: received 1 entries in 558.2 ms 11:16:53,558 AM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: persisted 1 entries in 33.82 μs 11:16:53,558 AM [follower344-writer-153] [DEBUG] JournalWriteTask - follower344: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@88f6182 stopped 11:16:53,558 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - follower344: journal closed 11:16:53,558 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - follower344: journal closed 11:16:53,558 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower344: stopped executor with 0 remaining tasks 11:16:53,558 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - follower344: stopped executor with 0 remaining tasks 11:16:53,559 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower345-notifier350#-1334034946] 11:16:53,559 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower345-collector351#1939096999] 11:16:53,559 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader342-notifier352#-496520754] 11:16:53,559 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader342-collector353#-1741508726] 11:16:53,560 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader342] 11:16:53,560 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] PekkoRaftStorage - leader342: terminating thread VirtualThread[#731,leader342-writer-155]/waiting 11:16:53,560 AM [leader342-writer-155] [DEBUG] JournalWriteTask - leader342: received 1 entries in 551.0 ms 11:16:53,560 AM [leader342-writer-155] [DEBUG] JournalWriteTask - leader342: persisted 1 entries in 19.78 μs 11:16:53,560 AM [leader342-writer-155] [DEBUG] JournalWriteTask - leader342: task org.opendaylight.controller.cluster.raft.spi.JournalWriteTask@6986601c stopped 11:16:53,560 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - leader342: journal closed 11:16:53,560 AM [test-pekko.actor.default-dispatcher-6] [INFO] PekkoRaftStorage - leader342: journal closed 11:16:53,560 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader342: stopped executor with 0 remaining tasks 11:16:53,560 AM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - leader342: stopped executor with 0 remaining tasks 11:16:53,561 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$d#-1100830818] [INFO] [06/27/2025 11:16:53.563] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 11:16:53,563 AM [test-pekko.actor.default-dispatcher-6] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [06/27/2025 11:16:53.564] [test-pekko.actor.default-dispatcher-18] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.170.168:17355) 11:16:53,565 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.168:17355] - Exiting completed 11:16:53,565 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.168:17355] - Shutting down... [INFO] [06/27/2025 11:16:53.564] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.168:17355] - Exiting completed 11:16:53,565 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.168:17355] - Successfully shut down [INFO] [06/27/2025 11:16:53.565] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.168:17355] - Shutting down... [INFO] [06/27/2025 11:16:53.565] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.168:17355] - Successfully shut down [INFO] [06/27/2025 11:16:53.566] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.170.168:17355/system/remoting-terminator] Shutting down remote daemon. 11:16:53,566 AM [test-pekko.actor.default-dispatcher-6] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. [INFO] [06/27/2025 11:16:53.566] [test-pekko.remote.default-remote-dispatcher-9] [pekko://test@10.30.170.168:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 11:16:53,567 AM [test-pekko.actor.default-dispatcher-18] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports.