[DEBUG] [01/30/2025 14:39:17.895] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 02:39:17,896 PM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [01/30/2025 14:39:17.896] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [01/30/2025 14:39:17.896] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [01/30/2025 14:39:17.908] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.145:17355] with UID [5016463755157284127] 02:39:17,908 PM [test-pekko.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.170.145:17355] with UID [5016463755157284127] [INFO] [01/30/2025 14:39:17.909] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.145:17355] - Starting up, Pekko version [1.0.3] ... 02:39:17,909 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.145:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [01/30/2025 14:39:17.912] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.145:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [01/30/2025 14:39:17.912] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.145:17355] - Started up successfully 02:39:17,912 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.145:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 02:39:17,912 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.145:17355] - Started up successfully [INFO] [01/30/2025 14:39:17.913] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.145:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 02:39:17,913 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.145:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [01/30/2025 14:39:17.913] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.145:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 02:39:17,913 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.145:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 02:39:17,923 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferAborted starting 02:39:17,923 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:39:17,928 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower550 with journal batch size 1000 [WARN] [01/30/2025 14:39:17.930] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower550]: org.opendaylight.controller.cluster.common.actor.Monitor@60a7abde [INFO] [pekkoDeadLetter][01/30/2025 14:39:17.928] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower550] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:17,930 PM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower550] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [01/30/2025 14:39:17.935] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower551]: org.opendaylight.controller.cluster.common.actor.Monitor@411de2da [INFO] [pekkoDeadLetter][01/30/2025 14:39:17.936] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower551] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:17,936 PM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower551] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:17,937 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower551 with journal batch size 1000 02:39:17,938 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: SnapshotOffer called. 02:39:17,938 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Moving last applied index from -1 to -1 02:39:17,938 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower551: Set currentTerm=1, votedFor=null 02:39:17,938 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Updated server config: isVoting: true, peers: [PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=NON_VOTING], PeerInfo [id=leader549, address=pekko://test/user/leader549, votingState=VOTING], PeerInfo [id=follower550, address=pekko://test/user/follower550, votingState=VOTING]] 02:39:17,938 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower551 in 101.6 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:17,938 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: SnapshotOffer called. 02:39:17,938 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: Moving last applied index from -1 to -1 02:39:17,938 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - follower550: Set currentTerm=1, votedFor=null 02:39:17,938 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: Updated server config: isVoting: true, peers: [PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=NON_VOTING], PeerInfo [id=leader549, address=pekko://test/user/leader549, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING]] [WARN] [01/30/2025 14:39:17.939] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower552]: org.opendaylight.controller.cluster.common.actor.Monitor@63d0156d [INFO] [pekkoDeadLetter][01/30/2025 14:39:17.939] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower552] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:17,938 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower550 in 227.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:17,939 PM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower552] to Actor[pekko://test/user/termination-monitor] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:17,939 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower552 with journal batch size 1000 02:39:17,940 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: Recovery completed in 2.182 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:17,940 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower550: Set currentTerm=1, votedFor=null 02:39:17,940 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: Local TermInfo store seeded with TermInfo{term=1} 02:39:17,940 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Recovery completed in 2.562 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:17,940 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: Snapshot capture initiated after recovery due to migrated messages 02:39:17,940 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - follower551: Set currentTerm=1, votedFor=null 02:39:17,940 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:17,940 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:17,940 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: lastSequenceNumber prior to capture: 0 02:39:17,940 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Local TermInfo store seeded with TermInfo{term=1} 02:39:17,941 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Snapshot capture initiated after recovery due to migrated messages 02:39:17,941 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:17,941 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:17,941 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: lastSequenceNumber prior to capture: 0 02:39:17,942 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: CaptureSnapshotReply received by actor 02:39:17,942 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower552, isVoting=false], ServerInfo[peerId=leader549, isVoting=true], ServerInfo[peerId=follower551, isVoting=true], ServerInfo[peerId=follower550, isVoting=true]]]] 02:39:17,942 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:17,942 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: CaptureSnapshotReply received by actor 02:39:17,942 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower552, isVoting=false], ServerInfo[peerId=leader549, isVoting=true], ServerInfo[peerId=follower550, isVoting=true], ServerInfo[peerId=follower551, isVoting=true]]]] 02:39:17,942 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: SnapshotOffer called. 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Moving last applied index from -1 to -1 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower552: Set currentTerm=1, votedFor=null 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Updated server config: isVoting: false, peers: [PeerInfo [id=leader549, address=pekko://test/user/leader549, votingState=VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING], PeerInfo [id=follower550, address=pekko://test/user/follower550, votingState=VOTING]] 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower552 in 48.75 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: Snapshot success - sequence number: -1 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: SnapshotComplete received 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: Snapshot success - sequence number: -1 02:39:17,943 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: SnapshotComplete received 02:39:17,943 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader549 with journal batch size 1000 02:39:17,944 PM [test-pekko.actor.default-dispatcher-6] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader549] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:17,944 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: handled a journal response 02:39:17,944 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Recovery completed in 1.178 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:17,944 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - follower552: Set currentTerm=1, votedFor=null 02:39:17,944 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Local TermInfo store seeded with TermInfo{term=1} 02:39:17,944 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Snapshot capture initiated after recovery due to migrated messages 02:39:17,944 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: SnapshotOffer called. 02:39:17,944 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:17,944 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Moving last applied index from -1 to -1 02:39:17,944 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - leader549: Set currentTerm=1, votedFor=null 02:39:17,944 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:17,944 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: lastSequenceNumber prior to capture: 0 02:39:17,944 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Updated server config: isVoting: true, peers: [PeerInfo [id=follower552, address=pekko://test/user/follower552, votingState=NON_VOTING], PeerInfo [id=follower551, address=pekko://test/user/follower551, votingState=VOTING], PeerInfo [id=follower550, address=pekko://test/user/follower550, votingState=VOTING]] 02:39:17,944 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader549 in 105.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:17,944 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: handled a snapshot response [WARN] [01/30/2025 14:39:17.944] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader549]: org.opendaylight.controller.cluster.common.actor.Monitor@7246e5ab [INFO] [pekkoDeadLetter][01/30/2025 14:39:17.943] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader549] to Actor[pekko://test/user/termination-monitor] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:17,945 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: handled a journal response 02:39:17,945 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: handled a snapshot response 02:39:17,945 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Recovery completed in 1.212 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:17,945 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - leader549: Set currentTerm=1, votedFor=null 02:39:17,945 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: CaptureSnapshotReply received by actor 02:39:17,945 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Local TermInfo store seeded with TermInfo{term=1} 02:39:17,945 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Snapshot capture initiated after recovery due to migrated messages 02:39:17,945 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:17,945 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:17,945 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: lastSequenceNumber prior to capture: 0 02:39:17,945 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=leader549, isVoting=true], ServerInfo[peerId=follower551, isVoting=true], ServerInfo[peerId=follower550, isVoting=true], ServerInfo[peerId=follower552, isVoting=false]]]] 02:39:17,945 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:17,946 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: CaptureSnapshotReply received by actor 02:39:17,946 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower552, isVoting=false], ServerInfo[peerId=follower551, isVoting=true], ServerInfo[peerId=follower550, isVoting=true], ServerInfo[peerId=leader549, isVoting=true]]]] 02:39:17,946 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:17,947 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:17,947 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:17,947 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Snapshot success - sequence number: -1 02:39:17,947 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: Snapshot success - sequence number: -1 02:39:17,947 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: SnapshotComplete received 02:39:17,947 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: SnapshotComplete received 02:39:17,947 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: handled a journal response 02:39:17,947 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: handled a journal response 02:39:17,947 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: handled a snapshot response 02:39:17,947 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: handled a snapshot response 02:39:18,334 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:39:18,334 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.145:17355, Removed) 02:39:18,334 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Follower): Received ElectionTimeout - switching to Candidate 02:39:18,334 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Candidate): Election: Candidate has following voting peers: [follower551, follower550] 02:39:18,334 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - leader549: Set currentTerm=2, votedFor=leader549 02:39:18,335 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Candidate): Starting new election term 2 02:39:18,335 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Candidate): Sending RequestVote{term=2, candidateId=leader549, lastLogIndex=-1, lastLogTerm=-1} to peer follower551 02:39:18,335 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Candidate): Sending RequestVote{term=2, candidateId=leader549, lastLogIndex=-1, lastLogTerm=-1} to peer follower550 02:39:18,335 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:18,335 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:39:18,335 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Cluster state: [] 02:39:18,335 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader549, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:18,335 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader549, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:18,335 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower551: Set currentTerm=2, votedFor=null 02:39:18,335 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:18,335 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550 (Follower): Cluster state: [] 02:39:18,335 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader549, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:18,335 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): In requestVote: RequestVote{term=2, candidateId=leader549, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:18,335 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower550 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader549, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:18,335 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower551: Set currentTerm=2, votedFor=leader549 02:39:18,335 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - follower550: Set currentTerm=2, votedFor=null 02:39:18,335 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:18,335 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550 (Follower): In requestVote: RequestVote{term=2, candidateId=leader549, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:18,335 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - follower550: Set currentTerm=2, votedFor=leader549 02:39:18,335 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:18,336 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:39:18,336 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:39:18,337 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Election: Leader has following peers: [follower552, follower551, follower550] 02:39:18,337 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Checking sendAppendEntries for follower follower552: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:18,337 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Checking sendAppendEntries for follower follower551: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:18,337 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Checking sendAppendEntries for follower follower550: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:18,337 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:39:18,337 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower552 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader549, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:39:18,337 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower552: Set currentTerm=2, votedFor=null 02:39:18,337 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower552: Last sync leader does not match current leader leader549, need to catch up to -1 02:39:18,337 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower552 updated: matchIndex: -1, nextIndex: 0 02:39:18,337 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower551: Last sync leader does not match current leader leader549, need to catch up to -1 02:39:18,337 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:18.337] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower551] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@412f088d 02:39:18,338 PM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:18,338 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower551 updated: matchIndex: -1, nextIndex: 0 02:39:18,338 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower550: Last sync leader does not match current leader leader549, need to catch up to -1 02:39:18,338 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:18,338 PM [test-pekko.actor.default-dispatcher-13] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:18,338 PM [test-pekko.actor.default-dispatcher-14] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower550] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:18,338 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower550 updated: matchIndex: -1, nextIndex: 0 [INFO] [pekkoDeadLetter][01/30/2025 14:39:18.337] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [01/30/2025 14:39:18.338] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower552] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@76942711 02:39:18,338 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Checking sendAppendEntries for follower follower550: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [INFO] [pekkoDeadLetter][01/30/2025 14:39:18.338] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [01/30/2025 14:39:18.338] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower550] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3396b7a7 [INFO] [pekkoDeadLetter][01/30/2025 14:39:18.338] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower550] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower550] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:18,349 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:39:18,349 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader549 starting 02:39:18,351 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: onShutDown 02:39:18,351 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: Initiating leader transfer 02:39:18,351 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorLeadershipTransferCohort - leader549: pauseLeader successfully completed - doing transfer 02:39:18,351 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Attempting to transfer leadership 02:39:18,351 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower551: onLeaderTransitioning: LeaderTransitioning [leaderId=leader549] 02:39:18,351 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower552: onLeaderTransitioning: LeaderTransitioning [leaderId=leader549] 02:39:18,352 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower550: onLeaderTransitioning: LeaderTransitioning [leaderId=leader549] 02:39:18,354 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Checking sendAppendEntries for follower follower552: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:18,354 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Checking sendAppendEntries for follower follower551: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:18,354 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Checking sendAppendEntries for follower follower550: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:18,354 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower550: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,354 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,354 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) [WARN] [01/30/2025 14:39:18.354] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower550] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3c6c1b4a [WARN] [01/30/2025 14:39:18.354] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower551] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7e636fa1 [INFO] [pekkoDeadLetter][01/30/2025 14:39:18.354] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower550] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower550] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:18,354 PM [test-pekko.actor.default-dispatcher-18] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower550] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][01/30/2025 14:39:18.354] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower551] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:18,355 PM [test-pekko.actor.default-dispatcher-18] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower551] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [01/30/2025 14:39:18.355] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower552] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@17e93b32 [INFO] [pekkoDeadLetter][01/30/2025 14:39:18.356] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower552] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:18,356 PM [test-pekko.actor.default-dispatcher-14] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower552] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 02:39:18,455 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,455 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower550: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,455 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,574 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,575 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower550: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,575 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,694 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] RaftActorLeadershipTransferCohort - leader549: leader transfer aborted 02:39:18,694 PM [test-pekko.actor.default-dispatcher-17] [WARN] RaftActorLeadershipTransferCohort - leader549: Failed to transfer leadership in 343.1 ms 02:39:18,694 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549: leader transfer failed - sending PoisonPill 02:39:18,694 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader549 (Leader): Leadership transfer expired 02:39:18,695 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower551: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,695 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower552: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,695 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower550: Lagging 0 entries behind leader leader549 and reached -1 (of expected -1) 02:39:18,697 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader549 ending 02:39:18,699 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 02:39:18,699 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower550-notifier553#1248841708] 02:39:18,711 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower550-collector554#-222778736] 02:39:18,711 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower550] 02:39:18,711 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower551-notifier555#519017025] 02:39:18,711 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower551-collector556#163209464] 02:39:18,712 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower551] 02:39:18,712 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower552-notifier557#763595974] 02:39:18,712 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower552-collector558#-1323485409] 02:39:18,712 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower552] 02:39:18,713 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader549-notifier559#872265605] 02:39:18,713 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader549-collector560#-1185667024] 02:39:18,713 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader549] 02:39:18,715 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 starting 02:39:18,715 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/30/2025 14:39:18.717] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower562]: org.opendaylight.controller.cluster.common.actor.Monitor@30d688ca 02:39:18,718 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower562 with journal batch size 1000 02:39:18,719 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: SnapshotOffer called. 02:39:18,719 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Moving last applied index from -1 to -1 02:39:18,719 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower562: Set currentTerm=1, votedFor=null 02:39:18,719 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Updated server config: isVoting: true, peers: [PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=NON_VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=leader561, address=pekko://test/user/leader561, votingState=VOTING]] 02:39:18,719 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower562 in 105.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:18,720 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Recovery completed in 994.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:18,720 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower562: Set currentTerm=1, votedFor=null 02:39:18,720 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Local TermInfo store seeded with TermInfo{term=1} 02:39:18,720 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Snapshot capture initiated after recovery due to migrated messages 02:39:18,720 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:18,720 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:18,720 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: lastSequenceNumber prior to capture: 0 02:39:18,721 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: CaptureSnapshotReply received by actor 02:39:18,721 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower564, isVoting=false], ServerInfo[peerId=follower563, isVoting=true], ServerInfo[peerId=leader561, isVoting=true], ServerInfo[peerId=follower562, isVoting=true]]]] 02:39:18,722 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:18,722 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:18,722 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: Snapshot success - sequence number: -1 02:39:18,722 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: SnapshotComplete received 02:39:18,722 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: handled a snapshot response 02:39:18,722 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: handled a journal response [WARN] [01/30/2025 14:39:18.820] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower563]: org.opendaylight.controller.cluster.common.actor.Monitor@7af708cd 02:39:18,821 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower563 with journal batch size 1000 02:39:18,822 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: SnapshotOffer called. 02:39:18,822 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Moving last applied index from -1 to -1 02:39:18,822 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower563: Set currentTerm=1, votedFor=null 02:39:18,822 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Updated server config: isVoting: true, peers: [PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=NON_VOTING], PeerInfo [id=leader561, address=pekko://test/user/leader561, votingState=VOTING], PeerInfo [id=follower562, address=pekko://test/user/follower562, votingState=VOTING]] 02:39:18,822 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower563 in 72.00 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:18,823 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Recovery completed in 1.152 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:18,823 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower563: Set currentTerm=1, votedFor=null 02:39:18,823 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Local TermInfo store seeded with TermInfo{term=1} 02:39:18,823 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Snapshot capture initiated after recovery due to migrated messages 02:39:18,823 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:18,823 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:18,823 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: lastSequenceNumber prior to capture: 0 [WARN] [01/30/2025 14:39:18.824] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower564]: org.opendaylight.controller.cluster.common.actor.Monitor@1da3f41a 02:39:18,825 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower564 with journal batch size 1000 02:39:18,826 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: SnapshotOffer called. 02:39:18,826 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Moving last applied index from -1 to -1 02:39:18,826 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower564: Set currentTerm=1, votedFor=null 02:39:18,826 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Updated server config: isVoting: false, peers: [PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=leader561, address=pekko://test/user/leader561, votingState=VOTING], PeerInfo [id=follower562, address=pekko://test/user/follower562, votingState=VOTING]] 02:39:18,826 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower564 in 158.5 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [01/30/2025 14:39:18.827] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader561]: org.opendaylight.controller.cluster.common.actor.Monitor@3bd4e3b8 02:39:18,827 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader561 with journal batch size 1000 02:39:18,828 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: SnapshotOffer called. 02:39:18,828 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Moving last applied index from -1 to -1 02:39:18,828 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - leader561: Set currentTerm=1, votedFor=null 02:39:18,828 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Updated server config: isVoting: true, peers: [PeerInfo [id=follower564, address=pekko://test/user/follower564, votingState=NON_VOTING], PeerInfo [id=follower563, address=pekko://test/user/follower563, votingState=VOTING], PeerInfo [id=follower562, address=pekko://test/user/follower562, votingState=VOTING]] 02:39:18,828 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Recovery completed in 2.551 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:18,828 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: CaptureSnapshotReply received by actor 02:39:18,828 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - follower564: Set currentTerm=1, votedFor=null 02:39:18,828 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader561 in 101.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:18,828 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Local TermInfo store seeded with TermInfo{term=1} 02:39:18,828 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Snapshot capture initiated after recovery due to migrated messages 02:39:18,829 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:18,828 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower564, isVoting=false], ServerInfo[peerId=leader561, isVoting=true], ServerInfo[peerId=follower562, isVoting=true], ServerInfo[peerId=follower563, isVoting=true]]]] 02:39:18,829 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:18,829 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:18,829 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: lastSequenceNumber prior to capture: 0 02:39:18,829 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:18,829 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: Snapshot success - sequence number: -1 02:39:18,830 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: SnapshotComplete received 02:39:18,830 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: handled a journal response 02:39:18,830 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Recovery completed in 1.829 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:18,830 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - leader561: Set currentTerm=1, votedFor=null 02:39:18,830 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: CaptureSnapshotReply received by actor 02:39:18,830 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Local TermInfo store seeded with TermInfo{term=1} 02:39:18,830 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Snapshot capture initiated after recovery due to migrated messages 02:39:18,830 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:18,830 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: handled a snapshot response 02:39:18,830 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower563, isVoting=true], ServerInfo[peerId=leader561, isVoting=true], ServerInfo[peerId=follower562, isVoting=true], ServerInfo[peerId=follower564, isVoting=false]]]] 02:39:18,830 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:18,830 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: lastSequenceNumber prior to capture: 0 02:39:18,830 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:18,830 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:18,830 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: Snapshot success - sequence number: -1 02:39:18,831 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: SnapshotComplete received 02:39:18,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: handled a snapshot response 02:39:18,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: handled a journal response 02:39:18,831 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: CaptureSnapshotReply received by actor 02:39:18,831 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower564, isVoting=false], ServerInfo[peerId=follower563, isVoting=true], ServerInfo[peerId=follower562, isVoting=true], ServerInfo[peerId=leader561, isVoting=true]]]] 02:39:18,831 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:18,832 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:18,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Snapshot success - sequence number: -1 02:39:18,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: SnapshotComplete received 02:39:18,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: handled a snapshot response 02:39:18,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: handled a journal response 02:39:19,224 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:39:19,224 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.145:17355, Removed) 02:39:19,224 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Follower): Received ElectionTimeout - switching to Candidate 02:39:19,224 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Candidate): Election: Candidate has following voting peers: [follower563, follower562] 02:39:19,224 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - leader561: Set currentTerm=2, votedFor=leader561 02:39:19,224 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Candidate): Starting new election term 2 02:39:19,224 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Candidate): Sending RequestVote{term=2, candidateId=leader561, lastLogIndex=-1, lastLogTerm=-1} to peer follower563 02:39:19,224 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Candidate): Sending RequestVote{term=2, candidateId=leader561, lastLogIndex=-1, lastLogTerm=-1} to peer follower562 02:39:19,225 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:39:19,225 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:19,225 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:19,225 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Cluster state: [] 02:39:19,225 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader561, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:19,225 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader561, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:19,225 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower562: Set currentTerm=2, votedFor=null 02:39:19,225 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Cluster state: [] 02:39:19,225 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader561, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:19,225 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader561, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:19,225 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower563: Set currentTerm=2, votedFor=null 02:39:19,225 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): In requestVote: RequestVote{term=2, candidateId=leader561, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:19,225 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower562: Set currentTerm=2, votedFor=leader561 02:39:19,225 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:19,225 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): In requestVote: RequestVote{term=2, candidateId=leader561, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:19,225 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower563: Set currentTerm=2, votedFor=leader561 02:39:19,225 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:19,226 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:39:19,226 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:39:19,226 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Election: Leader has following peers: [follower564, follower563, follower562] 02:39:19,226 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower564: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,226 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower563: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,226 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower562: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,226 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:39:19,227 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower563: Last sync leader does not match current leader leader561, need to catch up to -1 [WARN] [01/30/2025 14:39:19.227] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@423aac99 02:39:19,227 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower562: Last sync leader does not match current leader leader561, need to catch up to -1 02:39:19,227 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader561, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:39:19,227 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower563 updated: matchIndex: -1, nextIndex: 0 02:39:19,227 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,227 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower564: Set currentTerm=2, votedFor=null 02:39:19,227 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower562 updated: matchIndex: -1, nextIndex: 0 02:39:19,227 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,227 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower564: Last sync leader does not match current leader leader561, need to catch up to -1 02:39:19,228 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower564 updated: matchIndex: -1, nextIndex: 0 [WARN] [01/30/2025 14:39:19.227] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower562] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6517cc76 02:39:19,228 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:19.228] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@36c51e1 02:39:19,234 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:39:19,234 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 02:39:19,235 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 02:39:19,235 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 02:39:19,235 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 02:39:19,236 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: onRequestLeadership RequestLeadership [requestedFollowerId=follower563, replyTo=Actor[pekko://test/user/$a#1355207467]] 02:39:19,238 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Initiating leader transfer 02:39:19,238 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower564: onLeaderTransitioning: LeaderTransitioning [leaderId=leader561] 02:39:19,239 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562: onLeaderTransitioning: LeaderTransitioning [leaderId=leader561] 02:39:19,239 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563: onLeaderTransitioning: LeaderTransitioning [leaderId=leader561] 02:39:19,239 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorLeadershipTransferCohort - leader561: pauseLeader successfully completed - doing transfer 02:39:19,239 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Attempting to transfer leadership 02:39:19,239 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,239 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader561 and reached -1 (of expected -1) 02:39:19,239 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:19.239] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2c36c2b3 02:39:19,239 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,239 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader561 and reached -1 (of expected -1) 02:39:19,239 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower562: Lagging 0 entries behind leader leader561 and reached -1 (of expected -1) [WARN] [01/30/2025 14:39:19.240] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower562] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7288144 [WARN] [01/30/2025 14:39:19.240] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower563] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3ec7bf36 02:39:19,240 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): tryToCompleteLeadershipTransfer: followerId: follower563, matchIndex: -1, lastIndex: -1, isVoting: true 02:39:19,240 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Follower's log matches - sending ElectionTimeout 02:39:19,240 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,240 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower564: Lagging 0 entries behind leader leader561 and reached -1 (of expected -1) 02:39:19,240 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower563: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,241 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower563: Lagging 0 entries behind leader leader561 and reached -1 (of expected -1) 02:39:19,241 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Checking sendAppendEntries for follower follower562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,241 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower562: Lagging 0 entries behind leader leader561 and reached -1 (of expected -1) 02:39:19,241 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:39:19,241 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower): Received TimeoutNow - switching to Candidate 02:39:19,241 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Candidate): Election: Candidate has following voting peers: [leader561, follower562] 02:39:19,241 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower563: Set currentTerm=3, votedFor=follower563 02:39:19,241 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Candidate): Starting new election term 3 02:39:19,241 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Candidate): Sending RequestVote{term=3, candidateId=follower563, lastLogIndex=-1, lastLogTerm=-1} to peer leader561 02:39:19,241 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Candidate): Sending RequestVote{term=3, candidateId=follower563, lastLogIndex=-1, lastLogTerm=-1} to peer follower562 02:39:19,241 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 02:39:19,242 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Leader transfer complete 02:39:19,242 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] RaftActorLeadershipTransferCohort - leader561: leader transfer complete - waiting for new leader 02:39:19,242 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:19,242 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Cluster state: [] 02:39:19,242 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower563, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:19,242 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower563, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 2 - updating term 02:39:19,242 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower562: Set currentTerm=3, votedFor=null 02:39:19,242 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): In requestVote: RequestVote{term=3, candidateId=follower563, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:19,242 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower562: Set currentTerm=3, votedFor=follower563 02:39:19,242 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower562 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 02:39:19,242 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:19,243 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Cluster state: [] 02:39:19,243 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower563, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:19,243 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower563, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower 02:39:19,243 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - leader561: Set currentTerm=3, votedFor=null 02:39:19,243 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): Leadership transfer in progress - processing RequestVote 02:39:19,243 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): In requestVote: RequestVote{term=3, candidateId=follower563, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:19,243 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - leader561: Set currentTerm=3, votedFor=follower563 02:39:19,243 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 02:39:19,243 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader561 (Leader) :- Switching from behavior Leader to Follower, election term: 3 02:39:19,243 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 02:39:19,243 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 02:39:19,243 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Leader): Election: Leader has following peers: [follower564, leader561, follower562] 02:39:19,243 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Leader): Checking sendAppendEntries for follower follower564: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,244 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Leader): Checking sendAppendEntries for follower leader561: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,244 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Leader): Checking sendAppendEntries for follower follower562: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,244 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower564 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower563, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 02:39:19,244 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower564: Set currentTerm=3, votedFor=null 02:39:19,244 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 02:39:19,244 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower564: Last sync leader does not match current leader follower563, need to catch up to -1 02:39:19,244 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower564 updated: matchIndex: -1, nextIndex: 0 02:39:19,244 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower562: Last sync leader does not match current leader follower563, need to catch up to -1 02:39:19,244 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Leader): Checking sendAppendEntries for follower follower564: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:19.244] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower562] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@607e9353 [WARN] [01/30/2025 14:39:19.244] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower564] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@231f0cb9 02:39:19,244 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower562 updated: matchIndex: -1, nextIndex: 0 02:39:19,244 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Leader): Checking sendAppendEntries for follower follower562: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,244 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - leader561: Last sync leader does not match current leader follower563, need to catch up to -1 02:39:19,245 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader561: leader changed to follower563 02:39:19,245 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader561 updated: matchIndex: -1, nextIndex: 0 02:39:19,245 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower563 (Leader): Checking sendAppendEntries for follower leader561: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,245 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorLeadershipTransferCohort - leader561: Successfully transferred leadership to follower563 in 6.357 ms 02:39:19,245 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader561: Leadership transferred successfully to follower563 [WARN] [01/30/2025 14:39:19.245] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/leader561] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4a23009d 02:39:19,287 PM [main] [INFO] LeadershipTransferIntegrationTest - testSuccessfulRequestLeadershipTransferToFollower2 ending 02:39:19,287 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower562-notifier565#901984445] 02:39:19,298 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower562-collector566#1129652571] 02:39:19,299 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower562] 02:39:19,299 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower563-notifier567#-772299873] 02:39:19,299 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower563-collector568#-912575441] 02:39:19,300 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower563] 02:39:19,300 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower564-notifier569#1111004268] 02:39:19,300 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower564-collector570#93302509] 02:39:19,300 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower564] 02:39:19,300 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader561-notifier571#-1424901055] 02:39:19,301 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader561-collector572#-1786022771] 02:39:19,301 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader561] 02:39:19,301 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$a#1355207467] 02:39:19,302 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown starting 02:39:19,302 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/30/2025 14:39:19.305] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower574]: org.opendaylight.controller.cluster.common.actor.Monitor@100dcec 02:39:19,306 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower574 with journal batch size 1000 02:39:19,307 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: SnapshotOffer called. 02:39:19,307 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Moving last applied index from -1 to -1 02:39:19,307 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - follower574: Set currentTerm=1, votedFor=null 02:39:19,307 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Updated server config: isVoting: true, peers: [PeerInfo [id=leader573, address=pekko://test/user/leader573, votingState=VOTING], PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=NON_VOTING]] 02:39:19,307 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower574 in 87.80 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [01/30/2025 14:39:19.307] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower575]: org.opendaylight.controller.cluster.common.actor.Monitor@20a229e3 02:39:19,307 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower575 with journal batch size 1000 02:39:19,308 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Recovery completed in 1.121 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:19,308 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower574: Set currentTerm=1, votedFor=null 02:39:19,308 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Local TermInfo store seeded with TermInfo{term=1} 02:39:19,308 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Snapshot capture initiated after recovery due to migrated messages 02:39:19,308 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:19,308 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:19,308 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: lastSequenceNumber prior to capture: 0 02:39:19,309 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: SnapshotOffer called. 02:39:19,309 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Moving last applied index from -1 to -1 02:39:19,309 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower575: Set currentTerm=1, votedFor=null 02:39:19,310 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Updated server config: isVoting: true, peers: [PeerInfo [id=leader573, address=pekko://test/user/leader573, votingState=VOTING], PeerInfo [id=follower574, address=pekko://test/user/follower574, votingState=VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=NON_VOTING]] 02:39:19,310 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: CaptureSnapshotReply received by actor 02:39:19,310 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower575 in 487.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:19,310 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=leader573, isVoting=true], ServerInfo[peerId=follower575, isVoting=true], ServerInfo[peerId=follower576, isVoting=false], ServerInfo[peerId=follower574, isVoting=true]]]] 02:39:19,310 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:19,310 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:19,310 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: Snapshot success - sequence number: -1 02:39:19,310 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: SnapshotComplete received 02:39:19,310 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: handled a journal response 02:39:19,311 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: handled a snapshot response 02:39:19,311 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Recovery completed in 1.421 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:19,311 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower575: Set currentTerm=1, votedFor=null 02:39:19,311 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Local TermInfo store seeded with TermInfo{term=1} 02:39:19,311 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Snapshot capture initiated after recovery due to migrated messages 02:39:19,311 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:19,311 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:19,311 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: lastSequenceNumber prior to capture: 0 02:39:19,312 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: CaptureSnapshotReply received by actor 02:39:19,312 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=leader573, isVoting=true], ServerInfo[peerId=follower574, isVoting=true], ServerInfo[peerId=follower576, isVoting=false], ServerInfo[peerId=follower575, isVoting=true]]]] 02:39:19,312 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:19,312 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:19,312 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: Snapshot success - sequence number: -1 02:39:19,312 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: SnapshotComplete received 02:39:19,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: handled a snapshot response 02:39:19,312 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: handled a journal response [WARN] [01/30/2025 14:39:19.410] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower576]: org.opendaylight.controller.cluster.common.actor.Monitor@45393b01 02:39:19,412 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower576 with journal batch size 1000 02:39:19,412 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: SnapshotOffer called. 02:39:19,412 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Moving last applied index from -1 to -1 02:39:19,412 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower576: Set currentTerm=1, votedFor=null 02:39:19,412 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Updated server config: isVoting: false, peers: [PeerInfo [id=leader573, address=pekko://test/user/leader573, votingState=VOTING], PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower574, address=pekko://test/user/follower574, votingState=VOTING]] 02:39:19,412 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower576 in 75.84 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:19,413 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Recovery completed in 856.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:19,413 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower576: Set currentTerm=1, votedFor=null 02:39:19,413 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Local TermInfo store seeded with TermInfo{term=1} 02:39:19,413 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Snapshot capture initiated after recovery due to migrated messages 02:39:19,413 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:19,413 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:19,413 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: lastSequenceNumber prior to capture: 0 02:39:19,414 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: CaptureSnapshotReply received by actor 02:39:19,414 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=leader573, isVoting=true], ServerInfo[peerId=follower575, isVoting=true], ServerInfo[peerId=follower574, isVoting=true], ServerInfo[peerId=follower576, isVoting=false]]]] 02:39:19,414 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:19,415 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:19,415 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: Snapshot success - sequence number: -1 02:39:19,415 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: SnapshotComplete received 02:39:19,415 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: handled a snapshot response 02:39:19,415 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: handled a journal response [WARN] [01/30/2025 14:39:19.513] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader573]: org.opendaylight.controller.cluster.common.actor.Monitor@6617da20 02:39:19,513 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader573 with journal batch size 1000 02:39:19,514 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: SnapshotOffer called. 02:39:19,514 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Moving last applied index from -1 to -1 02:39:19,514 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - leader573: Set currentTerm=1, votedFor=null 02:39:19,514 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Updated server config: isVoting: true, peers: [PeerInfo [id=follower575, address=pekko://test/user/follower575, votingState=VOTING], PeerInfo [id=follower574, address=pekko://test/user/follower574, votingState=VOTING], PeerInfo [id=follower576, address=pekko://test/user/follower576, votingState=NON_VOTING]] 02:39:19,514 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader573 in 94.87 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:19,515 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Recovery completed in 878.0 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:19,515 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - leader573: Set currentTerm=1, votedFor=null 02:39:19,515 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Local TermInfo store seeded with TermInfo{term=1} 02:39:19,515 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Snapshot capture initiated after recovery due to migrated messages 02:39:19,515 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:19,515 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:19,515 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: lastSequenceNumber prior to capture: 0 02:39:19,516 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: CaptureSnapshotReply received by actor 02:39:19,516 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower575, isVoting=true], ServerInfo[peerId=follower574, isVoting=true], ServerInfo[peerId=follower576, isVoting=false], ServerInfo[peerId=leader573, isVoting=true]]]] 02:39:19,517 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:19,517 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:19,517 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Snapshot success - sequence number: -1 02:39:19,517 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: SnapshotComplete received 02:39:19,517 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: handled a snapshot response 02:39:19,517 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: handled a journal response 02:39:19,833 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:39:19,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.145:17355, Removed) 02:39:19,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Follower): Received ElectionTimeout - switching to Candidate 02:39:19,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Candidate): Election: Candidate has following voting peers: [follower575, follower574] 02:39:19,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - leader573: Set currentTerm=2, votedFor=leader573 02:39:19,834 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Candidate): Starting new election term 2 02:39:19,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Candidate): Sending RequestVote{term=2, candidateId=leader573, lastLogIndex=-1, lastLogTerm=-1} to peer follower575 02:39:19,834 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Candidate): Sending RequestVote{term=2, candidateId=leader573, lastLogIndex=-1, lastLogTerm=-1} to peer follower574 02:39:19,834 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:39:19,834 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:19,834 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:19,834 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574 (Follower): Cluster state: [] 02:39:19,834 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Cluster state: [] 02:39:19,834 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader573, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:19,834 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader573, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:19,834 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader573, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:19,834 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower574 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader573, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:19,834 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - follower575: Set currentTerm=2, votedFor=null 02:39:19,834 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower574: Set currentTerm=2, votedFor=null 02:39:19,835 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574 (Follower): In requestVote: RequestVote{term=2, candidateId=leader573, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:19,835 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower574: Set currentTerm=2, votedFor=leader573 02:39:19,835 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:19,835 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:39:19,835 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:39:19,835 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): In requestVote: RequestVote{term=2, candidateId=leader573, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:19,835 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - follower575: Set currentTerm=2, votedFor=leader573 02:39:19,835 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:19,835 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Election: Leader has following peers: [follower575, follower574, follower576] 02:39:19,835 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Checking sendAppendEntries for follower follower575: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,835 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Checking sendAppendEntries for follower follower574: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,835 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Checking sendAppendEntries for follower follower576: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,835 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:39:19,836 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower576 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader573, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:39:19,836 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower576: Set currentTerm=2, votedFor=null 02:39:19,836 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower576: Last sync leader does not match current leader leader573, need to catch up to -1 02:39:19,836 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower576 updated: matchIndex: -1, nextIndex: 0 02:39:19,836 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower574: Last sync leader does not match current leader leader573, need to catch up to -1 02:39:19,836 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,836 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower575: Last sync leader does not match current leader leader573, need to catch up to -1 02:39:19,836 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower574 updated: matchIndex: -1, nextIndex: 0 02:39:19,836 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Checking sendAppendEntries for follower follower574: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,836 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower575 updated: matchIndex: -1, nextIndex: 0 02:39:19,836 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:19.836] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower574] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7d9af778 [WARN] [01/30/2025 14:39:19.836] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower576] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@1c985273 [WARN] [01/30/2025 14:39:19.836] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower575] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6df68eb8 02:39:19,869 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:39:19,869 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 02:39:19,870 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 02:39:19,870 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower575 starting 02:39:19,870 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower575: onShutDown 02:39:19,871 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower575 ending 02:39:19,871 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 02:39:19,871 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 02:39:19,871 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: onRequestLeadership RequestLeadership [requestedFollowerId=follower575, replyTo=Actor[pekko://test/user/$b#-1011693782]] 02:39:19,871 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: Initiating leader transfer 02:39:19,871 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorLeadershipTransferCohort - leader573: pauseLeader successfully completed - doing transfer 02:39:19,871 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower574: onLeaderTransitioning: LeaderTransitioning [leaderId=leader573] 02:39:19,871 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Attempting to transfer leadership [WARN] [01/30/2025 14:39:19.871] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower575] received dead letter from TestActor[pekko://test/user/leader573]: LeaderTransitioning [leaderId=leader573] 02:39:19,872 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Checking sendAppendEntries for follower follower575: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,872 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Checking sendAppendEntries for follower follower574: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:19.872] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower575] received dead letter from TestActor[pekko://test/user/leader573]: AppendEntries{term=2, leaderId=leader573, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:39:19,872 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Checking sendAppendEntries for follower follower576: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:19,872 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower574: Lagging 0 entries behind leader leader573 and reached -1 (of expected -1) 02:39:19,872 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower576: Lagging 0 entries behind leader leader573 and reached -1 (of expected -1) 02:39:19,872 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower576: onLeaderTransitioning: LeaderTransitioning [leaderId=leader573] [WARN] [01/30/2025 14:39:19.872] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower574] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@e0eae37 [WARN] [01/30/2025 14:39:19.872] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower576] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3e67ec3c [WARN] [01/30/2025 14:39:19.954] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower575] received dead letter from TestActor[pekko://test/user/leader573]: AppendEntries{term=2, leaderId=leader573, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [01/30/2025 14:39:20.074] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower575] received dead letter from TestActor[pekko://test/user/leader573]: AppendEntries{term=2, leaderId=leader573, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:39:20,75 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower574: Lagging 0 entries behind leader leader573 and reached -1 (of expected -1) 02:39:20,75 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower576: Lagging 0 entries behind leader leader573 and reached -1 (of expected -1) 02:39:20,194 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorLeadershipTransferCohort - leader573: leader transfer aborted 02:39:20,194 PM [test-pekko.actor.default-dispatcher-6] [WARN] RaftActorLeadershipTransferCohort - leader573: Failed to transfer leadership in 322.8 ms 02:39:20,194 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573: LeadershipTransfer request from follower575 failed 02:39:20,195 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader573 (Leader): Leadership transfer expired [WARN] [01/30/2025 14:39:20.195] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower575] received dead letter from TestActor[pekko://test/user/leader573]: AppendEntries{term=2, leaderId=leader573, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:39:20,195 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower574: Lagging 0 entries behind leader leader573 and reached -1 (of expected -1) 02:39:20,195 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower576: Lagging 0 entries behind leader leader573 and reached -1 (of expected -1) 02:39:20,224 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Shutdown ending 02:39:20,224 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower574-notifier577#-1761614068] 02:39:20,235 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower574-collector578#-1365607414] 02:39:20,235 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower574] 02:39:20,237 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower575-notifier579#1900978437] 02:39:20,237 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower575-collector580#-137955813] 02:39:20,237 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower575] 02:39:20,238 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower576-notifier581#1700155435] 02:39:20,238 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower576-collector582#153309011] 02:39:20,238 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower576] 02:39:20,238 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader573-notifier583#-360946750] 02:39:20,238 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader573-collector584#-1653935943] 02:39:20,238 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader573] 02:39:20,239 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$b#-1011693782] 02:39:20,241 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown starting 02:39:20,241 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/30/2025 14:39:20.243] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower586]: org.opendaylight.controller.cluster.common.actor.Monitor@44ee943b 02:39:20,244 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower586 with journal batch size 1000 02:39:20,245 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: SnapshotOffer called. 02:39:20,245 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Moving last applied index from -1 to -1 02:39:20,245 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower586: Set currentTerm=1, votedFor=null 02:39:20,245 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Updated server config: isVoting: true, peers: [PeerInfo [id=leader585, address=pekko://test/user/leader585, votingState=VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=NON_VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 02:39:20,245 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower586 in 129.7 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:20,246 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Recovery completed in 974.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:20,246 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower586: Set currentTerm=1, votedFor=null 02:39:20,246 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Local TermInfo store seeded with TermInfo{term=1} 02:39:20,246 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Snapshot capture initiated after recovery due to migrated messages 02:39:20,246 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:20,246 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:20,246 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: lastSequenceNumber prior to capture: 0 [WARN] [01/30/2025 14:39:20.248] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower587]: org.opendaylight.controller.cluster.common.actor.Monitor@2ed5ff26 02:39:20,248 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: CaptureSnapshotReply received by actor 02:39:20,248 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower587 with journal batch size 1000 02:39:20,248 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=leader585, isVoting=true], ServerInfo[peerId=follower588, isVoting=false], ServerInfo[peerId=follower587, isVoting=true], ServerInfo[peerId=follower586, isVoting=true]]]] 02:39:20,248 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:20,249 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:20,249 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Snapshot success - sequence number: -1 02:39:20,249 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: SnapshotOffer called. 02:39:20,249 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to -1 02:39:20,249 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - follower587: Set currentTerm=1, votedFor=null 02:39:20,249 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: SnapshotComplete received 02:39:20,249 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Updated server config: isVoting: true, peers: [PeerInfo [id=leader585, address=pekko://test/user/leader585, votingState=VOTING], PeerInfo [id=follower586, address=pekko://test/user/follower586, votingState=VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=NON_VOTING]] 02:39:20,249 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower587 in 86.02 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:20,249 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: handled a snapshot response 02:39:20,249 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: handled a journal response 02:39:20,249 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Recovery completed in 630.5 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:20,249 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower587: Set currentTerm=1, votedFor=null 02:39:20,249 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Local TermInfo store seeded with TermInfo{term=1} 02:39:20,249 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Snapshot capture initiated after recovery due to migrated messages 02:39:20,249 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:20,249 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:20,250 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: lastSequenceNumber prior to capture: 0 02:39:20,250 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: CaptureSnapshotReply received by actor 02:39:20,250 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=leader585, isVoting=true], ServerInfo[peerId=follower586, isVoting=true], ServerInfo[peerId=follower588, isVoting=false], ServerInfo[peerId=follower587, isVoting=true]]]] 02:39:20,250 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:20,250 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:20,250 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Snapshot success - sequence number: -1 02:39:20,251 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: SnapshotComplete received 02:39:20,251 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: handled a snapshot response 02:39:20,251 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: handled a journal response [WARN] [01/30/2025 14:39:20.253] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower588]: org.opendaylight.controller.cluster.common.actor.Monitor@dcbb169 02:39:20,254 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower588 with journal batch size 1000 02:39:20,254 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: SnapshotOffer called. 02:39:20,254 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to -1 02:39:20,254 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower588: Set currentTerm=1, votedFor=null 02:39:20,254 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Updated server config: isVoting: false, peers: [PeerInfo [id=leader585, address=pekko://test/user/leader585, votingState=VOTING], PeerInfo [id=follower586, address=pekko://test/user/follower586, votingState=VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 02:39:20,255 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower588 in 107.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:20,255 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Recovery completed in 739.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:20,255 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - follower588: Set currentTerm=1, votedFor=null 02:39:20,255 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Local TermInfo store seeded with TermInfo{term=1} 02:39:20,255 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Snapshot capture initiated after recovery due to migrated messages 02:39:20,255 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:20,255 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:20,255 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: lastSequenceNumber prior to capture: 0 02:39:20,256 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: CaptureSnapshotReply received by actor 02:39:20,256 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=leader585, isVoting=true], ServerInfo[peerId=follower586, isVoting=true], ServerInfo[peerId=follower587, isVoting=true], ServerInfo[peerId=follower588, isVoting=false]]]] 02:39:20,256 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:20,256 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:20,256 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Snapshot success - sequence number: -1 02:39:20,256 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: SnapshotComplete received 02:39:20,256 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: handled a snapshot response 02:39:20,256 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: handled a journal response [WARN] [01/30/2025 14:39:20.256] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader585]: org.opendaylight.controller.cluster.common.actor.Monitor@3a712fda 02:39:20,257 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader585 with journal batch size 1000 02:39:20,257 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: SnapshotOffer called. 02:39:20,257 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Moving last applied index from -1 to -1 02:39:20,257 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - leader585: Set currentTerm=1, votedFor=null 02:39:20,257 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Updated server config: isVoting: true, peers: [PeerInfo [id=follower586, address=pekko://test/user/follower586, votingState=VOTING], PeerInfo [id=follower588, address=pekko://test/user/follower588, votingState=NON_VOTING], PeerInfo [id=follower587, address=pekko://test/user/follower587, votingState=VOTING]] 02:39:20,257 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader585 in 53.29 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:20,258 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Recovery completed in 523.8 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:20,258 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - leader585: Set currentTerm=1, votedFor=null 02:39:20,258 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Local TermInfo store seeded with TermInfo{term=1} 02:39:20,258 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Snapshot capture initiated after recovery due to migrated messages 02:39:20,258 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:20,258 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:20,258 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: lastSequenceNumber prior to capture: 0 02:39:20,259 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: CaptureSnapshotReply received by actor 02:39:20,259 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower586, isVoting=true], ServerInfo[peerId=follower588, isVoting=false], ServerInfo[peerId=follower587, isVoting=true], ServerInfo[peerId=leader585, isVoting=true]]]] 02:39:20,259 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:20,259 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:20,259 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Snapshot success - sequence number: -1 02:39:20,259 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: SnapshotComplete received 02:39:20,259 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: handled a snapshot response 02:39:20,259 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: handled a journal response 02:39:20,634 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:39:20,634 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.145:17355, Removed) 02:39:20,635 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Follower): Received ElectionTimeout - switching to Candidate 02:39:20,635 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Candidate): Election: Candidate has following voting peers: [follower586, follower587] 02:39:20,635 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - leader585: Set currentTerm=2, votedFor=leader585 02:39:20,635 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Candidate): Starting new election term 2 02:39:20,635 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Candidate): Sending RequestVote{term=2, candidateId=leader585, lastLogIndex=-1, lastLogTerm=-1} to peer follower586 02:39:20,635 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Candidate): Sending RequestVote{term=2, candidateId=leader585, lastLogIndex=-1, lastLogTerm=-1} to peer follower587 02:39:20,635 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:20,635 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:39:20,635 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): Cluster state: [] 02:39:20,635 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader585, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:20,635 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader585, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:20,635 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - follower586: Set currentTerm=2, votedFor=null 02:39:20,635 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): In requestVote: RequestVote{term=2, candidateId=leader585, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:20,635 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] PersistenceTermInfoStore - follower586: Set currentTerm=2, votedFor=leader585 02:39:20,635 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:20,636 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:20,636 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Cluster state: [] 02:39:20,636 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader585, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:20,636 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader585, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:20,636 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower587: Set currentTerm=2, votedFor=null 02:39:20,636 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): In requestVote: RequestVote{term=2, candidateId=leader585, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:20,636 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower587: Set currentTerm=2, votedFor=leader585 02:39:20,636 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:20,637 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:39:20,637 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:39:20,637 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Election: Leader has following peers: [follower586, follower588, follower587] 02:39:20,637 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower586: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:20,637 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:20,637 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower587: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:20,637 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:39:20,637 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader585, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:39:20,637 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - follower588: Set currentTerm=2, votedFor=null 02:39:20,637 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower587: Last sync leader does not match current leader leader585, need to catch up to -1 02:39:20,637 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader leader585, need to catch up to -1 02:39:20,637 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower586: Last sync leader does not match current leader leader585, need to catch up to -1 [WARN] [01/30/2025 14:39:20.637] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6e94bcc2 02:39:20,638 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: -1, nextIndex: 0 02:39:20,638 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:20,638 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: -1, nextIndex: 0 [WARN] [01/30/2025 14:39:20.638] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower586] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@211cf773 02:39:20,638 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:20,638 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower586 updated: matchIndex: -1, nextIndex: 0 02:39:20,638 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower586: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:20.638] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6bb92833 02:39:20,662 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:39:20,662 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower586: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): sendAppendEntries: 0 is present for follower follower586 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Sending AppendEntries to follower follower586: AppendEntries{term=2, leaderId=leader585, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): sendAppendEntries: 0 is present for follower follower588 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Sending AppendEntries to follower follower588: AppendEntries{term=2, leaderId=leader585, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): sendAppendEntries: 0 is present for follower follower587 02:39:20,662 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Sending AppendEntries to follower follower587: AppendEntries{term=2, leaderId=leader585, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:20,663 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader585, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:20,663 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Number of entries to be appended = 1 02:39:20,663 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 02:39:20,663 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Append entry to log zero 02:39:20,663 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:20,663 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Log size is now 1 02:39:20,663 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader585, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:20,663 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower588, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:39:20,663 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): Number of entries to be appended = 1 02:39:20,663 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader leader585 and reached -1 (of expected -1) 02:39:20,663 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 02:39:20,663 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): Append entry to log zero 02:39:20,663 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:20,663 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: 0, nextIndex: 1 02:39:20,663 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,663 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): Log size is now 1 02:39:20,663 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower586, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:39:20,663 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower586: Lagging 0 entries behind leader leader585 and reached -1 (of expected -1) 02:39:20,663 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower586 updated: matchIndex: 0, nextIndex: 1 02:39:20,663 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower586: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,664 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:20,664 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:20,664 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 02:39:20,664 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [01/30/2025 14:39:20.664] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@63510451 [WARN] [01/30/2025 14:39:20.664] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower586] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@63f4ba06 02:39:20,664 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Setting last applied to 0 02:39:20,664 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Moving last applied index from -1 to 0 02:39:20,664 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Applying state for log index 0 data zero 02:39:20,664 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: applyState called: zero 02:39:20,664 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Persisting ApplyJournalEntries with index=0 02:39:20,754 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): sendAppendEntries: 0 is present for follower follower587 02:39:20,754 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Sending AppendEntries to follower follower587: AppendEntries{term=2, leaderId=leader585, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:20,874 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): sendAppendEntries: 0 is present for follower follower587 02:39:20,874 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Sending AppendEntries to follower follower587: AppendEntries{term=2, leaderId=leader585, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:20,874 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Commit index set to 0 02:39:20,874 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): Commit index set to 0 02:39:20,874 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader leader585 and reached 0 (of expected -1) 02:39:20,874 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower586: Lagging 0 entries behind leader leader585 and reached 0 (of expected -1) 02:39:20,874 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 02:39:20,874 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 02:39:20,874 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Setting last applied to 0 02:39:20,874 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): Setting last applied to 0 02:39:20,874 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Moving last applied index from -1 to 0 02:39:20,874 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Moving last applied index from -1 to 0 02:39:20,874 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Applying state for log index 0 data zero 02:39:20,874 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Applying state for log index 0 data zero 02:39:20,875 PM [test-pekko.actor.default-dispatcher-17] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: applyState called: zero 02:39:20,875 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: applyState called: zero 02:39:20,875 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: Persisting ApplyJournalEntries with index=0 02:39:20,875 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: Persisting ApplyJournalEntries with index=0 02:39:20,914 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 02:39:20,915 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 starting 02:39:20,915 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: onShutDown 02:39:20,915 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: Initiating leader transfer 02:39:20,915 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader585: pauseLeader successfully completed - doing transfer 02:39:20,915 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Attempting to transfer leadership 02:39:20,915 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower586: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,915 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586: onLeaderTransitioning: LeaderTransitioning [leaderId=leader585] 02:39:20,915 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,915 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,915 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): sendAppendEntries: 0 is present for follower follower587 02:39:20,915 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower588: onLeaderTransitioning: LeaderTransitioning [leaderId=leader585] 02:39:20,915 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower586: Lagging 0 entries behind leader leader585 and reached 0 (of expected -1) 02:39:20,916 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader leader585 and reached 0 (of expected -1) 02:39:20,916 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): tryToCompleteLeadershipTransfer: followerId: follower586, matchIndex: 0, lastIndex: 0, isVoting: true 02:39:20,916 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Follower's log matches - sending ElectionTimeout 02:39:20,916 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower586: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,916 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,916 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,916 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): sendAppendEntries: 0 is present for follower follower587 02:39:20,916 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader leader585 and reached 0 (of expected -1) 02:39:20,916 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Leader transfer complete 02:39:20,916 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader585: leader transfer complete - waiting for new leader 02:39:20,916 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower586: Lagging 0 entries behind leader leader585 and reached 0 (of expected -1) 02:39:20,916 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:39:20,916 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower): Received TimeoutNow - switching to Candidate 02:39:20,916 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Candidate): Election: Candidate has following voting peers: [leader585, follower587] 02:39:20,917 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower586: Set currentTerm=3, votedFor=follower586 02:39:20,917 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Candidate): Starting new election term 3 02:39:20,917 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Candidate): Sending RequestVote{term=3, candidateId=follower586, lastLogIndex=0, lastLogTerm=2} to peer leader585 02:39:20,917 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Candidate): Sending RequestVote{term=3, candidateId=follower586, lastLogIndex=0, lastLogTerm=2} to peer follower587 02:39:20,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:20,917 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:20,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Cluster state: [] 02:39:20,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower586, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 02:39:20,917 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Cluster state: [] 02:39:20,917 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower586, lastLogIndex=0, lastLogTerm=2}" message is greater than leader's term 2 - switching to Follower 02:39:20,917 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=follower586, lastLogIndex=0, lastLogTerm=2} with higher term appears reachable, updating term. 02:39:20,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - leader585: Set currentTerm=3, votedFor=null 02:39:20,917 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Term 3 in "RequestVote{term=3, candidateId=follower586, lastLogIndex=0, lastLogTerm=2}" message is greater than follower's term 2 - updating term 02:39:20,917 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower587: Set currentTerm=3, votedFor=null 02:39:20,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): Leadership transfer in progress - processing RequestVote 02:39:20,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): In requestVote: RequestVote{term=3, candidateId=follower586, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 2 02:39:20,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - leader585: Set currentTerm=3, votedFor=follower586 02:39:20,917 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): In requestVote: RequestVote{term=3, candidateId=follower586, lastLogIndex=0, lastLogTerm=2} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:20,917 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower587: Set currentTerm=3, votedFor=follower586 02:39:20,917 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 02:39:20,917 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 02:39:20,917 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader585 (Leader) :- Switching from behavior Leader to Follower, election term: 3 02:39:20,917 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 02:39:20,918 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 02:39:20,918 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 02:39:20,918 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): Election: Leader has following peers: [leader585, follower588, follower587] 02:39:20,918 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): Checking sendAppendEntries for follower leader585: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,919 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): Checking sendAppendEntries for follower follower588: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,919 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - leader585: Last sync leader does not match current leader follower586, need to catch up to -1 02:39:20,919 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower588 (Follower): Term 3 in "AppendEntries{term=3, leaderId=follower586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 2 - updating term 02:39:20,919 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PersistenceTermInfoStore - follower588: Set currentTerm=3, votedFor=null 02:39:20,919 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorLeadershipTransferCohort - leader585: leader changed to follower586 02:39:20,919 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower588: Last sync leader does not match current leader follower586, need to catch up to -1 02:39:20,919 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorLeadershipTransferCohort - leader585: Successfully transferred leadership to follower586 in 3.686 ms 02:39:20,919 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader585: leader transfer succeeded - sending PoisonPill [WARN] [01/30/2025 14:39:20.919] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/leader585] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6b7a0e15 02:39:20,919 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): Checking sendAppendEntries for follower follower587: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,919 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [01/30/2025 14:39:20.919] [test-pekko.actor.default-dispatcher-17] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@5093e569 02:39:20,920 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader585 updated: matchIndex: 0, nextIndex: 1 02:39:20,920 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): Checking sendAppendEntries for follower leader585: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:20.920] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/leader585] received dead letter from TestActor[pekko://test/user/follower586]: AppendEntries{term=3, leaderId=follower586, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:39:20,920 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower588 updated: matchIndex: 0, nextIndex: 1 02:39:20,920 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): Checking sendAppendEntries for follower follower588: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:20,920 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader follower586 and reached 0 (of expected -1) [WARN] [01/30/2025 14:39:20.921] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower588] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7eaa5faf 02:39:21,434 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower588: Lagging 0 entries behind leader follower586 and reached 0 (of expected -1) [WARN] [01/30/2025 14:39:21.434] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/leader585] received dead letter from TestActor[pekko://test/user/follower586]: AppendEntries{term=3, leaderId=follower586, prevLogIndex=0, prevLogTerm=2, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:39:21,435 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower587: Last sync leader does not match current leader follower586, need to catch up to -1 02:39:21,435 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): sendAppendEntries: 0 is present for follower follower587 02:39:21,435 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): Sending AppendEntries to follower follower587: AppendEntries{term=3, leaderId=follower586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} [WARN] [01/30/2025 14:39:21.435] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3e9e78c6 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=follower586, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Number of entries to be appended = 1 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Append entry to log zero 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Log size is now 1 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Commit index set to 0 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=follower587, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:39:21,436 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower587 updated: matchIndex: 0, nextIndex: 1 02:39:21,436 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower586 (Leader): Checking sendAppendEntries for follower follower587: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower587: Lagging 0 entries behind leader follower586 and reached 0 (of expected -1) 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587 (Follower): Setting last applied to 0 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Moving last applied index from -1 to 0 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Applying state for log index 0 data zero 02:39:21,436 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: applyState called: zero 02:39:21,437 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:21,437 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: Persisting ApplyJournalEntries with index=0 [WARN] [01/30/2025 14:39:21.437] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower587] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@79a2d42e 02:39:21,473 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: onLeaderTransitioning: LeaderTransitioning [leaderId=leader585] 02:39:21,474 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDownToLeaderAndVerifyLeadershipTransferToFollower1 ending 02:39:21,474 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower587 starting 02:39:21,474 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower587: onShutDown 02:39:21,475 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/follower587 ending 02:39:21,475 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferOnShutDown ending 02:39:21,475 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower586-notifier589#160853970] 02:39:21,485 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower586-collector590#-1265486297] 02:39:21,485 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower586] 02:39:21,486 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587-notifier591#191228307] 02:39:21,486 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587-collector592#-871760634] 02:39:21,486 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower587] 02:39:21,486 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower588-notifier593#606446534] 02:39:21,486 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower588-collector594#2132553773] 02:39:21,486 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower588] 02:39:21,487 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader585-notifier595#-2090099931] 02:39:21,487 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader585-collector596#667428488] 02:39:21,487 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader585] 02:39:21,488 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers starting [WARN] [01/30/2025 14:39:21.489] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader597]: org.opendaylight.controller.cluster.common.actor.Monitor@43fae6e6 02:39:21,490 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader597 with journal batch size 1000 02:39:21,492 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: Recovery completed - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:21,492 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - leader597: Set currentTerm=0, votedFor=null 02:39:21,492 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: Local TermInfo store seeded with TermInfo{term=0} 02:39:21,492 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: Snapshot capture initiated after recovery due to migrated messages 02:39:21,492 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:21,492 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:21,492 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: lastSequenceNumber prior to capture: 0 02:39:21,493 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: CaptureSnapshotReply received by actor 02:39:21,493 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=0, electionVotedFor=null, ServerConfigPayload=null] 02:39:21,493 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:21,493 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:39:21,493 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597 (Follower): Received TimeoutNow - switching to Candidate 02:39:21,493 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597 (Candidate): Election: Candidate has following voting peers: [] 02:39:21,493 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - leader597: Set currentTerm=1, votedFor=leader597 02:39:21,493 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader597 (Candidate): Starting new election term 1 02:39:21,493 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader597 (Follower) :- Switching from behavior Follower to Candidate, election term: 1 02:39:21,494 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597 (Candidate): Received ElectionTimeout 02:39:21,494 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597 (Leader): Election: Leader has following peers: [] 02:39:21,494 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader597 (Candidate) :- Switching from behavior Candidate to Leader, election term: 1 02:39:21,494 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:21,494 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: Snapshot success - sequence number: -1 02:39:21,494 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: SnapshotComplete received 02:39:21,494 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: handled a snapshot response 02:39:21,494 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: handled a journal response 02:39:21,543 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader597 starting 02:39:21,544 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader597: onShutDown 02:39:21,544 PM [main] [INFO] LeadershipTransferIntegrationTest - sendShutDown for pekko://test/user/leader597 ending 02:39:21,544 PM [main] [INFO] LeadershipTransferIntegrationTest - testLeaderTransferSkippedOnShutdownWithNoFollowers ending 02:39:21,544 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader597-collector601#-1626730345] 02:39:21,555 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader597] 02:39:21,556 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging starting 02:39:21,556 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/30/2025 14:39:21.558] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower603]: org.opendaylight.controller.cluster.common.actor.Monitor@7341f07b 02:39:21,559 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower603 with journal batch size 1000 02:39:21,559 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: SnapshotOffer called. 02:39:21,559 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Moving last applied index from -1 to -1 02:39:21,559 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower603: Set currentTerm=1, votedFor=null 02:39:21,559 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=NON_VOTING], PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=leader602, address=pekko://test/user/leader602, votingState=VOTING]] 02:39:21,559 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower603 in 130.2 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:21,561 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Recovery completed in 1.134 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:21,561 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - follower603: Set currentTerm=1, votedFor=null 02:39:21,561 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Local TermInfo store seeded with TermInfo{term=1} 02:39:21,561 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Snapshot capture initiated after recovery due to migrated messages 02:39:21,561 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:21,561 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:21,561 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: lastSequenceNumber prior to capture: 0 02:39:21,562 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: CaptureSnapshotReply received by actor [WARN] [01/30/2025 14:39:21.562] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower604]: org.opendaylight.controller.cluster.common.actor.Monitor@7fd983e 02:39:21,562 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower605, isVoting=false], ServerInfo[peerId=follower604, isVoting=true], ServerInfo[peerId=leader602, isVoting=true], ServerInfo[peerId=follower603, isVoting=true]]]] 02:39:21,562 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:21,562 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:21,562 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Snapshot success - sequence number: -1 02:39:21,562 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: SnapshotComplete received 02:39:21,563 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: handled a snapshot response 02:39:21,563 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: handled a journal response 02:39:21,563 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower604 with journal batch size 1000 02:39:21,564 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: SnapshotOffer called. 02:39:21,564 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Moving last applied index from -1 to -1 02:39:21,564 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower604: Set currentTerm=1, votedFor=null 02:39:21,564 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=NON_VOTING], PeerInfo [id=leader602, address=pekko://test/user/leader602, votingState=VOTING], PeerInfo [id=follower603, address=pekko://test/user/follower603, votingState=VOTING]] 02:39:21,564 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower604 in 85.32 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:21,564 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Recovery completed in 723.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:21,565 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PersistenceTermInfoStore - follower604: Set currentTerm=1, votedFor=null 02:39:21,565 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Local TermInfo store seeded with TermInfo{term=1} 02:39:21,565 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Snapshot capture initiated after recovery due to migrated messages 02:39:21,565 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:21,565 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:21,565 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: lastSequenceNumber prior to capture: 0 [WARN] [01/30/2025 14:39:21.565] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower605]: org.opendaylight.controller.cluster.common.actor.Monitor@716a68d9 02:39:21,565 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower605 with journal batch size 1000 02:39:21,566 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: CaptureSnapshotReply received by actor 02:39:21,566 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower605, isVoting=false], ServerInfo[peerId=leader602, isVoting=true], ServerInfo[peerId=follower603, isVoting=true], ServerInfo[peerId=follower604, isVoting=true]]]] 02:39:21,566 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:21,566 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: SnapshotOffer called. 02:39:21,566 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:21,566 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: Snapshot success - sequence number: -1 02:39:21,566 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Moving last applied index from -1 to -1 02:39:21,566 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower605: Set currentTerm=1, votedFor=null 02:39:21,566 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: SnapshotComplete received 02:39:21,566 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: handled a journal response 02:39:21,566 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Updated server config: isVoting: false, peers: [PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=leader602, address=pekko://test/user/leader602, votingState=VOTING], PeerInfo [id=follower603, address=pekko://test/user/follower603, votingState=VOTING]] 02:39:21,566 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: handled a snapshot response 02:39:21,566 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower605 in 229.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 [WARN] [01/30/2025 14:39:21.567] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader602]: org.opendaylight.controller.cluster.common.actor.Monitor@1af9644e 02:39:21,567 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Recovery completed in 774.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:21,567 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower605: Set currentTerm=1, votedFor=null 02:39:21,567 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Local TermInfo store seeded with TermInfo{term=1} 02:39:21,567 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Snapshot capture initiated after recovery due to migrated messages 02:39:21,567 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:21,567 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:21,567 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: lastSequenceNumber prior to capture: 0 02:39:21,567 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader602 with journal batch size 1000 02:39:21,568 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: SnapshotOffer called. 02:39:21,568 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Moving last applied index from -1 to -1 02:39:21,568 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - leader602: Set currentTerm=1, votedFor=null 02:39:21,568 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Updated server config: isVoting: true, peers: [PeerInfo [id=follower605, address=pekko://test/user/follower605, votingState=NON_VOTING], PeerInfo [id=follower604, address=pekko://test/user/follower604, votingState=VOTING], PeerInfo [id=follower603, address=pekko://test/user/follower603, votingState=VOTING]] 02:39:21,568 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader602 in 64.24 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:21,568 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Recovery completed in 493.9 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:21,568 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - leader602: Set currentTerm=1, votedFor=null 02:39:21,568 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Local TermInfo store seeded with TermInfo{term=1} 02:39:21,568 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Snapshot capture initiated after recovery due to migrated messages 02:39:21,568 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:21,568 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:21,568 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: lastSequenceNumber prior to capture: 0 02:39:21,569 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: CaptureSnapshotReply received by actor 02:39:21,569 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: CaptureSnapshotReply received by actor 02:39:21,569 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower604, isVoting=true], ServerInfo[peerId=leader602, isVoting=true], ServerInfo[peerId=follower603, isVoting=true], ServerInfo[peerId=follower605, isVoting=false]]]] 02:39:21,569 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:21,569 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower605, isVoting=false], ServerInfo[peerId=follower604, isVoting=true], ServerInfo[peerId=follower603, isVoting=true], ServerInfo[peerId=leader602, isVoting=true]]]] 02:39:21,569 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:21,569 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:21,569 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Snapshot success - sequence number: -1 02:39:21,569 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:21,569 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Snapshot success - sequence number: -1 02:39:21,569 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: SnapshotComplete received 02:39:21,569 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: handled a journal response 02:39:21,569 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: handled a snapshot response 02:39:21,570 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: SnapshotComplete received 02:39:21,570 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: handled a journal response 02:39:21,570 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: handled a snapshot response 02:39:21,914 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:39:21,914 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.145:17355, Removed) 02:39:21,914 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Follower): Received ElectionTimeout - switching to Candidate 02:39:21,915 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Candidate): Election: Candidate has following voting peers: [follower604, follower603] 02:39:21,915 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - leader602: Set currentTerm=2, votedFor=leader602 02:39:21,915 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Candidate): Starting new election term 2 02:39:21,915 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Candidate): Sending RequestVote{term=2, candidateId=leader602, lastLogIndex=-1, lastLogTerm=-1} to peer follower604 02:39:21,915 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Candidate): Sending RequestVote{term=2, candidateId=leader602, lastLogIndex=-1, lastLogTerm=-1} to peer follower603 02:39:21,915 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:21,915 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:39:21,915 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Cluster state: [] 02:39:21,915 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:21,915 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader602, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:21,915 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader602, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:21,915 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower604: Set currentTerm=2, votedFor=null 02:39:21,915 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): Cluster state: [] 02:39:21,915 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader602, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:21,915 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader602, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:21,915 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - follower603: Set currentTerm=2, votedFor=null 02:39:21,915 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): In requestVote: RequestVote{term=2, candidateId=leader602, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:21,915 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower604: Set currentTerm=2, votedFor=leader602 02:39:21,915 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): In requestVote: RequestVote{term=2, candidateId=leader602, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:21,915 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - follower603: Set currentTerm=2, votedFor=leader602 02:39:21,915 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:21,915 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:21,917 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:39:21,917 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:39:21,917 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Election: Leader has following peers: [follower605, follower604, follower603] 02:39:21,917 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower605: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:21,917 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower604: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:21,917 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower603: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:21,917 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader602, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:39:21,917 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - follower605: Set currentTerm=2, votedFor=null 02:39:21,917 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:39:21,917 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower604: Last sync leader does not match current leader leader602, need to catch up to -1 02:39:21,917 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower605: Last sync leader does not match current leader leader602, need to catch up to -1 02:39:21,917 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower603: Last sync leader does not match current leader leader602, need to catch up to -1 02:39:21,917 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower604 updated: matchIndex: -1, nextIndex: 0 02:39:21,917 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:21.917] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower603] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3d5272e2 02:39:21,918 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower605 updated: matchIndex: -1, nextIndex: 0 [WARN] [01/30/2025 14:39:21.918] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower605] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@6cd8aa12 02:39:21,918 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:21.918] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/follower604] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@2f30fc8d 02:39:21,918 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower603 updated: matchIndex: -1, nextIndex: 0 02:39:21,918 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower603: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:21,921 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:39:21,921 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 02:39:21,922 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 02:39:21,922 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging starting 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Persist data SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Replicate message: identifier: MockIdentifier{value=zero}, logIndex: 0, isSendImmediate: true 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): sendAppendEntries: 0 is present for follower follower605 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Sending AppendEntries to follower follower605: AppendEntries{term=2, leaderId=leader602, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): sendAppendEntries: 0 is present for follower follower604 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Sending AppendEntries to follower follower604: AppendEntries{term=2, leaderId=leader602, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower603: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): sendAppendEntries: 0 is present for follower follower603 02:39:21,922 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Sending AppendEntries to follower follower603: AppendEntries{term=2, leaderId=leader602, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:21,922 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader602, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:21,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Number of entries to be appended = 1 02:39:21,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 02:39:21,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Append entry to log zero 02:39:21,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:21,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=leader602, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:21,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Log size is now 1 02:39:21,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): Number of entries to be appended = 1 02:39:21,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 02:39:21,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): Append entry to log zero 02:39:21,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower605, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:39:21,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Append log entry and persist SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:21,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower605: Lagging 0 entries behind leader leader602 and reached -1 (of expected -1) 02:39:21,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): Log size is now 1 02:39:21,923 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower605 updated: matchIndex: 0, nextIndex: 1 02:39:21,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=follower603, logLastIndex=0, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 02:39:21,923 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower603: Lagging 0 entries behind leader leader602 and reached -1 (of expected -1) 02:39:21,923 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:21,923 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower603 updated: matchIndex: 0, nextIndex: 1 02:39:21,923 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower603: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:21,923 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:21,923 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 02:39:21,923 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Setting last applied to 0 02:39:21,923 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Moving last applied index from -1 to 0 02:39:21,923 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Applying state for log index 0 data zero 02:39:21,923 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: applyState called: zero 02:39:21,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] 02:39:21,923 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: persist complete SimpleReplicatedLogEntry [index=0, term=2, payload=zero] [WARN] [01/30/2025 14:39:21.923] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower603] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@50c8565d [WARN] [01/30/2025 14:39:21.923] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower605] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7cb7d330 02:39:21,924 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Persisting ApplyJournalEntries with index=0 02:39:22,34 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): sendAppendEntries: 0 is present for follower follower604 02:39:22,34 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Sending AppendEntries to follower follower604: AppendEntries{term=2, leaderId=leader602, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:22,34 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Commit index set to 0 02:39:22,34 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): Commit index set to 0 02:39:22,34 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower605: Lagging 0 entries behind leader leader602 and reached 0 (of expected -1) 02:39:22,34 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower603: Lagging 0 entries behind leader leader602 and reached 0 (of expected -1) 02:39:22,34 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 02:39:22,34 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 02:39:22,34 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605 (Follower): Setting last applied to 0 02:39:22,34 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603 (Follower): Setting last applied to 0 02:39:22,34 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Moving last applied index from -1 to 0 02:39:22,34 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Moving last applied index from -1 to 0 02:39:22,34 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Applying state for log index 0 data zero 02:39:22,34 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Applying state for log index 0 data zero 02:39:22,34 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: applyState called: zero 02:39:22,34 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: applyState called: zero 02:39:22,35 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: Persisting ApplyJournalEntries with index=0 02:39:22,35 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: Persisting ApplyJournalEntries with index=0 02:39:22,73 PM [main] [INFO] LeadershipTransferIntegrationTest - sendPayloadWithFollower2Lagging ending 02:39:22,74 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 02:39:22,74 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 02:39:22,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: onRequestLeadership RequestLeadership [requestedFollowerId=follower604, replyTo=Actor[pekko://test/user/$c#1302692483]] 02:39:22,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: Initiating leader transfer 02:39:22,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader602: pauseLeader successfully completed - doing transfer 02:39:22,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Attempting to transfer leadership 02:39:22,74 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower605: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:22,74 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower604: onLeaderTransitioning: LeaderTransitioning [leaderId=leader602] 02:39:22,74 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower605: onLeaderTransitioning: LeaderTransitioning [leaderId=leader602] 02:39:22,74 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower603: onLeaderTransitioning: LeaderTransitioning [leaderId=leader602] 02:39:22,75 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower604: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:22,75 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): sendAppendEntries: 0 is present for follower follower604 02:39:22,75 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Checking sendAppendEntries for follower follower603: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 02:39:22,75 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower603: Lagging 0 entries behind leader leader602 and reached 0 (of expected -1) 02:39:22,75 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower605: Lagging 0 entries behind leader leader602 and reached 0 (of expected -1) 02:39:22,154 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): sendAppendEntries: 0 is present for follower follower604 02:39:22,154 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Sending AppendEntries to follower follower604: AppendEntries{term=2, leaderId=leader602, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]} 02:39:22,274 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] SyncStatusTracker - follower605: Lagging 0 entries behind leader leader602 and reached 0 (of expected -1) 02:39:22,274 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower603: Lagging 0 entries behind leader leader602 and reached 0 (of expected -1) 02:39:22,394 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] RaftActorLeadershipTransferCohort - leader602: leader transfer aborted 02:39:22,394 PM [test-pekko.actor.default-dispatcher-18] [WARN] RaftActorLeadershipTransferCohort - leader602: Failed to transfer leadership in 320.1 ms 02:39:22,394 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602: LeadershipTransfer request from follower604 failed 02:39:22,394 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader602 (Leader): Leadership transfer expired 02:39:22,395 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower603: Lagging 0 entries behind leader leader602 and reached 0 (of expected -1) 02:39:22,395 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - follower605: Lagging 0 entries behind leader leader602 and reached 0 (of expected -1) 02:39:22,427 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithFollower2Lagging ending 02:39:22,427 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower603-notifier606#-144356644] 02:39:22,438 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower603-collector607#1086798960] 02:39:22,438 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower603] 02:39:22,439 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower604-notifier608#1723954723] 02:39:22,439 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower604-collector609#1738179583] 02:39:22,439 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower604] 02:39:22,439 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower605-notifier610#1944112672] 02:39:22,439 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower605-collector611#-1477951369] 02:39:22,440 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower605] 02:39:22,440 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader602-notifier612#-1343123263] 02:39:22,440 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader602-collector613#-1377586495] 02:39:22,440 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader602] 02:39:22,440 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$c#1302692483] 02:39:22,441 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown starting 02:39:22,442 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting [WARN] [01/30/2025 14:39:22.443] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower615]: org.opendaylight.controller.cluster.common.actor.Monitor@6cbc9855 02:39:22,444 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower615 with journal batch size 1000 02:39:22,445 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: SnapshotOffer called. 02:39:22,445 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: Moving last applied index from -1 to -1 02:39:22,445 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower615: Set currentTerm=1, votedFor=null 02:39:22,445 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=NON_VOTING], PeerInfo [id=leader614, address=pekko://test/user/leader614, votingState=VOTING]] 02:39:22,445 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower615 in 116.1 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:22,446 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: Recovery completed in 1.300 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:22,446 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - follower615: Set currentTerm=1, votedFor=null 02:39:22,447 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: Local TermInfo store seeded with TermInfo{term=1} 02:39:22,447 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: Snapshot capture initiated after recovery due to migrated messages 02:39:22,447 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. [WARN] [01/30/2025 14:39:22.447] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower616]: org.opendaylight.controller.cluster.common.actor.Monitor@4df41ded 02:39:22,447 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:22,447 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: lastSequenceNumber prior to capture: 0 02:39:22,447 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower616 with journal batch size 1000 02:39:22,448 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: SnapshotOffer called. 02:39:22,448 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Moving last applied index from -1 to -1 02:39:22,448 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower616: Set currentTerm=1, votedFor=null 02:39:22,448 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Updated server config: isVoting: true, peers: [PeerInfo [id=follower615, address=pekko://test/user/follower615, votingState=VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=NON_VOTING], PeerInfo [id=leader614, address=pekko://test/user/leader614, votingState=VOTING]] 02:39:22,448 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower616 in 49.90 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:22,448 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: CaptureSnapshotReply received by actor 02:39:22,448 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower616, isVoting=true], ServerInfo[peerId=follower617, isVoting=false], ServerInfo[peerId=leader614, isVoting=true], ServerInfo[peerId=follower615, isVoting=true]]]] 02:39:22,448 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:22,449 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:22,449 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: Snapshot success - sequence number: -1 02:39:22,449 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Recovery completed in 1.072 ms - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:22,449 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower616: Set currentTerm=1, votedFor=null 02:39:22,449 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Local TermInfo store seeded with TermInfo{term=1} 02:39:22,449 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Snapshot capture initiated after recovery due to migrated messages 02:39:22,449 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:22,449 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:22,449 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: lastSequenceNumber prior to capture: 0 [WARN] [01/30/2025 14:39:22.449] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower617]: org.opendaylight.controller.cluster.common.actor.Monitor@6b2aadfc 02:39:22,449 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for follower617 with journal batch size 1000 02:39:22,450 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: CaptureSnapshotReply received by actor 02:39:22,450 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: SnapshotComplete received 02:39:22,450 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: handled a journal response 02:39:22,450 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower615, isVoting=true], ServerInfo[peerId=follower617, isVoting=false], ServerInfo[peerId=leader614, isVoting=true], ServerInfo[peerId=follower616, isVoting=true]]]] 02:39:22,450 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:22,450 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: SnapshotOffer called. 02:39:22,450 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Moving last applied index from -1 to -1 02:39:22,450 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower617: Set currentTerm=1, votedFor=null 02:39:22,450 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615: handled a snapshot response 02:39:22,450 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:22,450 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: Snapshot success - sequence number: -1 02:39:22,450 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: SnapshotComplete received 02:39:22,450 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: handled a snapshot response 02:39:22,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: handled a journal response 02:39:22,451 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Updated server config: isVoting: false, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower615, address=pekko://test/user/follower615, votingState=VOTING], PeerInfo [id=leader614, address=pekko://test/user/leader614, votingState=VOTING]] 02:39:22,451 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for follower617 in 299.9 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:22,451 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Recovery completed in 718.2 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:22,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - follower617: Set currentTerm=1, votedFor=null 02:39:22,451 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Local TermInfo store seeded with TermInfo{term=1} 02:39:22,451 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Snapshot capture initiated after recovery due to migrated messages 02:39:22,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:22,451 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:22,451 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: lastSequenceNumber prior to capture: 0 02:39:22,452 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: CaptureSnapshotReply received by actor 02:39:22,452 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower616, isVoting=true], ServerInfo[peerId=follower615, isVoting=true], ServerInfo[peerId=leader614, isVoting=true], ServerInfo[peerId=follower617, isVoting=false]]]] 02:39:22,452 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:22,452 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:22,452 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: Snapshot success - sequence number: -1 02:39:22,452 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: SnapshotComplete received 02:39:22,452 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: handled a snapshot response 02:39:22,452 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower617: handled a journal response [WARN] [01/30/2025 14:39:22.452] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader614]: org.opendaylight.controller.cluster.common.actor.Monitor@8baa0b3 02:39:22,453 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Starting recovery for leader614 with journal batch size 1000 02:39:22,453 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: SnapshotOffer called. 02:39:22,453 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Moving last applied index from -1 to -1 02:39:22,453 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - leader614: Set currentTerm=1, votedFor=null 02:39:22,453 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Updated server config: isVoting: true, peers: [PeerInfo [id=follower616, address=pekko://test/user/follower616, votingState=VOTING], PeerInfo [id=follower615, address=pekko://test/user/follower615, votingState=VOTING], PeerInfo [id=follower617, address=pekko://test/user/follower617, votingState=NON_VOTING]] 02:39:22,453 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - Recovery snapshot applied for leader614 in 45.92 μs: snapshotIndex=-1, snapshotTerm=-1, journal-size=0 02:39:22,453 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Recovery completed in 438.3 μs - Switching actor to Follower - last log index = -1, last log term = -1, snapshot index = -1, snapshot term = -1, journal size = 0 02:39:22,453 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - leader614: Set currentTerm=1, votedFor=null 02:39:22,453 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Local TermInfo store seeded with TermInfo{term=1} 02:39:22,453 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Snapshot capture initiated after recovery due to migrated messages 02:39:22,453 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 02:39:22,453 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] 02:39:22,454 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: lastSequenceNumber prior to capture: 0 02:39:22,454 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: CaptureSnapshotReply received by actor 02:39:22,454 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Persisting of snapshot done: Snapshot [lastIndex=-1, lastTerm=-1, lastAppliedIndex=-1, lastAppliedTerm=-1, unAppliedEntries size=0, state=MockSnapshotState [state=[]], electionTerm=1, electionVotedFor=null, ServerConfigPayload=ServerConfigurationPayload [serverConfig=[ServerInfo[peerId=follower616, isVoting=true], ServerInfo[peerId=follower615, isVoting=true], ServerInfo[peerId=follower617, isVoting=false], ServerInfo[peerId=leader614, isVoting=true]]]] 02:39:22,454 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Removed in-memory snapshotted entries, adjusted snaphsotIndex: -1 and term: -1 02:39:22,454 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: SaveSnapshotSuccess received for snapshot, sequenceNr: -1 02:39:22,454 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Snapshot success - sequence number: -1 02:39:22,454 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: SnapshotComplete received 02:39:22,455 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: handled a snapshot response 02:39:22,455 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: handled a journal response 02:39:22,874 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:39:22,875 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Follower): Checking if this node is isolated in the cluster unreachable set [],all members [] self member: Member(pekko://test@10.30.170.145:17355, Removed) 02:39:22,875 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Follower): Received ElectionTimeout - switching to Candidate 02:39:22,875 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Candidate): Election: Candidate has following voting peers: [follower616, follower615] 02:39:22,875 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] PersistenceTermInfoStore - leader614: Set currentTerm=2, votedFor=leader614 02:39:22,875 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Candidate): Starting new election term 2 02:39:22,875 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Candidate): Sending RequestVote{term=2, candidateId=leader614, lastLogIndex=-1, lastLogTerm=-1} to peer follower616 02:39:22,875 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Candidate): Sending RequestVote{term=2, candidateId=leader614, lastLogIndex=-1, lastLogTerm=-1} to peer follower615 02:39:22,875 PM [test-pekko.actor.default-dispatcher-14] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 02:39:22,875 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:22,875 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Cluster state: [] 02:39:22,875 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader614, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:22,875 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:22,875 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader614, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:22,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower616: Set currentTerm=2, votedFor=null 02:39:22,876 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615 (Follower): Cluster state: [] 02:39:22,876 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=leader614, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:22,876 PM [test-pekko.actor.default-dispatcher-18] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower615 (Follower): Term 2 in "RequestVote{term=2, candidateId=leader614, lastLogIndex=-1, lastLogTerm=-1}" message is greater than follower's term 1 - updating term 02:39:22,876 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower615: Set currentTerm=2, votedFor=null 02:39:22,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): In requestVote: RequestVote{term=2, candidateId=leader614, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:22,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - follower616: Set currentTerm=2, votedFor=leader614 02:39:22,876 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:22,876 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615 (Follower): In requestVote: RequestVote{term=2, candidateId=leader614, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 2, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:22,876 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] PersistenceTermInfoStore - follower615: Set currentTerm=2, votedFor=leader614 02:39:22,876 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower615 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 02:39:22,877 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 02:39:22,877 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 02:39:22,877 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Election: Leader has following peers: [follower616, follower615, follower617] 02:39:22,877 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower616: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,878 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower615: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,878 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower617: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,878 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 02:39:22,878 PM [test-pekko.actor.default-dispatcher-6] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower617 (Follower): Term 2 in "AppendEntries{term=2, leaderId=leader614, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 1 - updating term 02:39:22,878 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PersistenceTermInfoStore - follower617: Set currentTerm=2, votedFor=null 02:39:22,878 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower616: Last sync leader does not match current leader leader614, need to catch up to -1 02:39:22,878 PM [test-pekko.actor.default-dispatcher-18] [DEBUG] SyncStatusTracker - follower615: Last sync leader does not match current leader leader614, need to catch up to -1 02:39:22,878 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower617: Last sync leader does not match current leader leader614, need to catch up to -1 [WARN] [01/30/2025 14:39:22.878] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower616] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@426081e0 [WARN] [01/30/2025 14:39:22.878] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower617] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@3965db4f [WARN] [01/30/2025 14:39:22.878] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower615] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@7a103bb3 02:39:22,879 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower616 updated: matchIndex: -1, nextIndex: 0 02:39:22,879 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower615 updated: matchIndex: -1, nextIndex: 0 02:39:22,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower615: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower617 updated: matchIndex: -1, nextIndex: 0 02:39:22,879 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,907 PM [main] [INFO] LeadershipTransferIntegrationTest - createRaftActors starting 02:39:22,907 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor starting 02:39:22,907 PM [main] [INFO] LeadershipTransferIntegrationTest - createRequestLeadershipResultCollectorActor ending 02:39:22,907 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower615] 02:39:22,918 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower617] 02:39:22,929 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader starting 02:39:22,929 PM [main] [INFO] LeadershipTransferIntegrationTest - sendFollower2RequestLeadershipTransferToLeader ending 02:39:22,929 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: onRequestLeadership RequestLeadership [requestedFollowerId=follower616, replyTo=Actor[pekko://test/user/$d#-974395709]] 02:39:22,929 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Initiating leader transfer 02:39:22,929 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616: onLeaderTransitioning: LeaderTransitioning [leaderId=leader614] [WARN] [01/30/2025 14:39:22.929] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower615] received dead letter from TestActor[pekko://test/user/leader614]: LeaderTransitioning [leaderId=leader614] [WARN] [01/30/2025 14:39:22.929] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower617] received dead letter from TestActor[pekko://test/user/leader614]: LeaderTransitioning [leaderId=leader614] 02:39:22,930 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader614: pauseLeader successfully completed - doing transfer 02:39:22,930 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Attempting to transfer leadership 02:39:22,930 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,930 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower616: Lagging 0 entries behind leader leader614 and reached -1 (of expected -1) [WARN] [01/30/2025 14:39:22.930] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower616] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@4c234d0d 02:39:22,930 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower615: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,930 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,930 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): tryToCompleteLeadershipTransfer: followerId: follower616, matchIndex: -1, lastIndex: -1, isVoting: true 02:39:22,930 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Follower's log matches - sending ElectionTimeout 02:39:22,930 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower616: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:22.930] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower615] received dead letter from TestActor[pekko://test/user/leader614]: AppendEntries{term=2, leaderId=leader614, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [01/30/2025 14:39:22.930] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower617] received dead letter from TestActor[pekko://test/user/leader614]: AppendEntries{term=2, leaderId=leader614, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:39:22,930 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower616: Lagging 0 entries behind leader leader614 and reached -1 (of expected -1) 02:39:22,930 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower615: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,930 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Checking sendAppendEntries for follower follower617: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [01/30/2025 14:39:22.930] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower615] received dead letter from TestActor[pekko://test/user/leader614]: AppendEntries{term=2, leaderId=leader614, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} [WARN] [01/30/2025 14:39:22.930] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower617] received dead letter from TestActor[pekko://test/user/leader614]: AppendEntries{term=2, leaderId=leader614, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]} 02:39:22,930 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 02:39:22,930 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower): Received TimeoutNow - switching to Candidate 02:39:22,930 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Candidate): Election: Candidate has following voting peers: [follower615, leader614] 02:39:22,930 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PersistenceTermInfoStore - follower616: Set currentTerm=3, votedFor=follower616 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Candidate): Starting new election term 3 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Candidate): Sending RequestVote{term=3, candidateId=follower616, lastLogIndex=-1, lastLogTerm=-1} to peer follower615 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Candidate): Sending RequestVote{term=3, candidateId=follower616, lastLogIndex=-1, lastLogTerm=-1} to peer leader614 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 [WARN] [01/30/2025 14:39:22.931] [test-pekko.actor.default-dispatcher-18] [pekko://test/user/follower615] received dead letter from TestActor[pekko://test/user/follower616]: RequestVote{term=3, candidateId=follower616, lastLogIndex=-1, lastLogTerm=-1} 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Leader transfer complete 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader614: leader transfer complete - waiting for new leader 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Cluster state: [] 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Candidate in requestVote:RequestVote{term=3, candidateId=follower616, lastLogIndex=-1, lastLogTerm=-1} with higher term appears reachable, updating term. 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Term 3 in "RequestVote{term=3, candidateId=follower616, lastLogIndex=-1, lastLogTerm=-1}" message is greater than leader's term 2 - switching to Follower 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - leader614: Set currentTerm=3, votedFor=null 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): Leadership transfer in progress - processing RequestVote 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): In requestVote: RequestVote{term=3, candidateId=follower616, lastLogIndex=-1, lastLogTerm=-1} - currentTerm: 3, votedFor: null, lastIndex: -1, lastTerm: -1 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PersistenceTermInfoStore - leader614: Set currentTerm=3, votedFor=follower616 02:39:22,931 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Leader): Election: Leader has following peers: [follower615, follower617, leader614] 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Leader): Checking sendAppendEntries for follower follower615: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Leader): Checking sendAppendEntries for follower follower617: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Leader): Checking sendAppendEntries for follower leader614: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,931 PM [test-pekko.actor.default-dispatcher-12] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 [WARN] [01/30/2025 14:39:22.931] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower615] received dead letter from TestActor[pekko://test/user/follower616]: AppendEntries{term=3, leaderId=follower616, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 02:39:22,932 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractRaftActorIntegrationTest$TestRaftActor - leader614 (Leader) :- Switching from behavior Leader to Follower, election term: 3 [WARN] [01/30/2025 14:39:22.932] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower617] received dead letter from TestActor[pekko://test/user/follower616]: AppendEntries{term=3, leaderId=follower616, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 02:39:22,932 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - leader614: Last sync leader does not match current leader follower616, need to catch up to -1 02:39:22,932 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader614: leader changed to follower616 02:39:22,932 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Leader): handleAppendEntriesReply - FollowerLogInformation for leader614 updated: matchIndex: -1, nextIndex: 0 02:39:22,932 PM [test-pekko.actor.default-dispatcher-17] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - follower616 (Leader): Checking sendAppendEntries for follower leader614: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 02:39:22,932 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorLeadershipTransferCohort - leader614: Successfully transferred leadership to follower616 in 2.998 ms 02:39:22,932 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractRaftActorIntegrationTest$TestRaftActor - leader614: Leadership transferred successfully to follower616 [WARN] [01/30/2025 14:39:22.932] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/leader614] unhandled message from Actor[pekko://test/deadLetters]: org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus@75599974 02:39:22,980 PM [main] [INFO] LeadershipTransferIntegrationTest - testRequestLeadershipTransferToFollower2WithOtherFollowersDown ending 02:39:22,980 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower615-notifier618#1446439922] 02:39:22,991 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower615-collector619#-453711225] 02:39:22,991 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower616-notifier620#98793007] 02:39:22,991 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower616-collector621#-1340141040] 02:39:22,991 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower616] 02:39:22,992 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower617-notifier622#694071228] 02:39:22,992 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower617-collector623#-139525123] 02:39:22,992 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader614-notifier624#2065232427] 02:39:22,992 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader614-collector625#615598477] 02:39:22,992 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader614] 02:39:22,992 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/$d#-974395709] [INFO] [01/30/2025 14:39:22.993] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] 02:39:22,994 PM [test-pekko.actor.default-dispatcher-17] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [01/30/2025 14:39:22.995] [test-pekko.actor.default-dispatcher-17] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.170.145:17355) [INFO] [01/30/2025 14:39:22.995] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.145:17355] - Exiting completed [INFO] [01/30/2025 14:39:22.995] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.145:17355] - Shutting down... [INFO] [01/30/2025 14:39:22.996] [test-pekko.actor.internal-dispatcher-2] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.170.145:17355] - Successfully shut down 02:39:22,997 PM [test-pekko.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.145:17355] - Exiting completed 02:39:22,997 PM [test-pekko.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.145:17355] - Shutting down... 02:39:22,997 PM [test-pekko.actor.default-dispatcher-14] [INFO] Cluster - Cluster Node [pekko://test@10.30.170.145:17355] - Successfully shut down [INFO] [01/30/2025 14:39:22.997] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.170.145:17355/system/remoting-terminator] Shutting down remote daemon. 02:39:22,997 PM [test-pekko.actor.default-dispatcher-17] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. [INFO] [01/30/2025 14:39:22.997] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.170.145:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 02:39:22,997 PM [test-pekko.actor.default-dispatcher-14] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [01/30/2025 14:39:23.000] [test-pekko.remote.default-remote-dispatcher-7] [pekko://test@10.30.170.145:17355/system/remoting-terminator] Remoting shut down. 02:39:23,0 PM [test-pekko.actor.default-dispatcher-17] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down.