[DEBUG] [09/04/2025 13:21:30.549] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 01:21:30,552 PM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [09/04/2025 13:21:30.552] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [09/04/2025 13:21:30.552] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [09/04/2025 13:21:30.573] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.38:17355] with UID [8908702635135945655] 01:21:30,573 PM [test-pekko.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.38:17355] with UID [8908702635135945655] [INFO] [09/04/2025 13:21:30.574] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.38:17355] - Starting up, Pekko version [1.1.5] ... 01:21:30,574 PM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.38:17355] - Starting up, Pekko version [1.1.5] ... [INFO] [09/04/2025 13:21:30.577] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.38:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [09/04/2025 13:21:30.577] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.38:17355] - Started up successfully 01:21:30,577 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.38:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 01:21:30,577 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.38:17355] - Started up successfully [INFO] [09/04/2025 13:21:30.585] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.38:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [09/04/2025 13:21:30.585] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.38:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 01:21:30,586 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.38:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 01:21:30,586 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.38:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 01:21:30,595 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithOperationInProgress starting 01:21:30,596 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server109] 01:21:30,597 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:30,597 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - new-server: started executor 01:21:30,598 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - new-server: started executor 01:21:30,598 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:30,598 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:30,598 PM [test-pekko.actor.default-dispatcher-6] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:30,598 PM [new-server-writer-35] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:30,598 PM [new-server-writer-35] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:30,599 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:30,599 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - /tmp/junit-5755105613448154436/new-server/TermInfo.properties does not exist [WARN] [09/04/2025 13:21:30.599] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server109]: org.opendaylight.controller.cluster.common.actor.Monitor@7097adc9 01:21:30,600 PM [test-pekko.actor.default-dispatcher-11] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/new-server109] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][09/04/2025 13:21:30.599] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/new-server109] to Actor[pekko://test/user/termination-monitor] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:30,601 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:30,601 PM [test-pekko.actor.default-dispatcher-6] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:30,601 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/TermInfo.properties 01:21:30,601 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/journal-v1-1.log 01:21:30,601 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/metadata-v1-1.log 01:21:30,601 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:30,601 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:30,601 PM [test-pekko.actor.default-dispatcher-6] [INFO] Recovery - new-server: Recovery completed in in 2.551 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:30,601 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:30,602 PM [test-pekko.actor.default-dispatcher-6] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:30,607 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader110] [WARN] [09/04/2025 13:21:30.607] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader110]: org.opendaylight.controller.cluster.common.actor.Monitor@733d9f3a [INFO] [pekkoDeadLetter][09/04/2025 13:21:30.608] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader110] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:30,608 PM [test-pekko.actor.default-dispatcher-5] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader110] to Actor[pekko://test/user/termination-monitor] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:30,608 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:30,608 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:30,609 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: started executor 01:21:30,609 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: started executor 01:21:30,629 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:30,630 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:30,630 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:30,630 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:30,630 PM [leader-writer-36] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:30,630 PM [leader-writer-36] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:30,630 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/junit-5755105613448154436/leader/TermInfo.properties does not exist 01:21:30,635 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:30,635 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:30,635 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/TermInfo.properties 01:21:30,635 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/journal-v1-1.log 01:21:30,635 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/metadata-v1-1.log 01:21:30,635 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: no eligible files found 01:21:30,635 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:30,636 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - leader: Recovery completed in in 4.878 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:30,636 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:30,636 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [] 01:21:30,646 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server109, votingMember=true], state: Idle 01:21:30,670 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server109, votingMember=true] 01:21:30,671 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 01:21:30,671 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on new-server 01:21:30,671 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader: last applied journal index prior to capture: 0 01:21:30,671 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader: support() called 01:21:30,673 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating capture snapshot for new server new-server 01:21:30,674 PM [leader-%d863157] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:30,676 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:30,676 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader: Persising snapshot at EntryInfo[index=-1, term=-1]/EntryInfo[index=-1, term=-1] 01:21:30,676 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader: support() called 01:21:30,676 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 01:21:30,676 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): sendInstallSnapshot 01:21:30,676 PM [leader-%d863158] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2852E4ED.tmp 01:21:30,677 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Snapshot 157 bytes, total chunks to send: 1 01:21:30,677 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Next chunk: total length=157, offset=0, size=157, hashCode=1546036366 01:21:30,677 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): next snapshot chunk size for follower new-server: 157 01:21:30,677 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): InstallSnapshot sent to follower Vector(user, new-server109), Chunk: 1/1 01:21:30,678 PM [leader-%d863158] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2852E4ED.v1 01:21:30,679 PM [leader-%d863158] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2852E4ED.v1 01:21:30,679 PM [leader-%d863158] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/TermInfo.properties 01:21:30,679 PM [leader-%d863158] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/journal-v1-1.log 01:21:30,679 PM [leader-%d863158] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/metadata-v1-1.log 01:21:30,679 PM [leader-%d863158] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:30.676521197Z, file=/tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2852E4ED.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:30,679 PM [leader-%d863158] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:30,679 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:30,679 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader: snapshot is durable as of 2025-09-04T13:21:30.676521197Z 01:21:30,679 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - leader: Snapshot success, discarding journal entries up to 0 01:21:30,679 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorSnapshotMessageSupport - leader: SnapshotComplete received 01:21:30,709 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server2, newServerAddress=pekko://test/user/follower107, votingMember=false], state: InstallingSnapshot 01:21:30,709 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Server operation already in progress - queueing AddServer [newServerId=new-server2, newServerAddress=pekko://test/user/follower107, votingMember=false] 01:21:30,710 PM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - new-server (Follower): Term 1 in "InstallSnapshot{term=1, leaderId=leader, lastIncludedIndex=-1, lastIncludedTerm=-1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1]}" message is greater than follower's term 0 - updating term 01:21:30,717 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=1, votedFor=null 01:21:30,717 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): handleInstallSnapshot: InstallSnapshot{term=1, leaderId=leader, lastIncludedIndex=-1, lastIncludedTerm=-1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1]} 01:21:30,718 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - new-server: Last sync leader does not match current leader leader, need to catch up to -1 01:21:30,718 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotTracker - new-server (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 01:21:30,718 PM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - new-server (Follower): Snapshot received from leader: leader 01:21:30,718 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - new-server: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=157}} 01:21:30,718 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - new-server: support() called [WARN] [09/04/2025 13:21:30.718] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/new-server109] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=false] [INFO] [pekkoDeadLetter][09/04/2025 13:21:30.718] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/new-server109] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/new-server109] was unhandled. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:30,718 PM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/new-server109] was unhandled. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:30,719 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - new-server: Converted InstallSnapshot from leader: leader to state 01:21:30,719 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - new-server: last applied journal index prior to persisting applied snapshot: 0 01:21:30,719 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - new-server: support() called 01:21:30,720 PM [new-server-%d311904] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2AEB2B62.tmp 01:21:30,722 PM [new-server-%d311904] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2AEB2B62.v1 01:21:30,722 PM [new-server-%d311904] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2AEB2B62.v1 01:21:30,722 PM [new-server-%d311904] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/TermInfo.properties 01:21:30,722 PM [new-server-%d311904] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/journal-v1-1.log 01:21:30,722 PM [new-server-%d311904] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/metadata-v1-1.log 01:21:30,722 PM [new-server-%d311904] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:30.720055138Z, file=/tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2AEB2B62.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:30,722 PM [new-server-%d311904] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:30,722 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:30,722 PM [test-pekko.actor.default-dispatcher-6] [INFO] SnapshotManager - new-server: snapshot is durable as of 2025-09-04T13:21:30.720055138Z 01:21:30,723 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - new-server: support() called 01:21:30,723 PM [test-pekko.actor.default-dispatcher-6] [INFO] MockRaftActor - new-server: applySnapshot called 01:21:30,723 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Follower - new-server (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=1, followerId=new-server, chunkIndex=1, success=true} 01:21:30,723 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SnapshotManager - new-server: Snapshot success, discarding journal entries up to 0 01:21:30,723 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] RaftActorSnapshotMessageSupport - new-server: SnapshotComplete received 01:21:30,723 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=1, followerId=new-server, chunkIndex=1, success=true} 01:21:30,723 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractLeader - leader (Leader): Snapshot successfully installed on follower new-server (last chunk 1) - matchIndex set to -1, nextIndex set to 0 01:21:30,724 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - Sent message UnInitializedFollowerSnapshotReply to self 01:21:30,724 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: onUnInitializedFollowerSnapshotReply: UnInitializedFollowerSnapshotReply [followerId=new-server] 01:21:30,724 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]] 01:21:30,724 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Persist data index=0 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:30,724 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:30,725 PM [leader-%d863159] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2B36D259.tmp 01:21:30,725 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=7c9bb05f-f0f1-40cd-9e3b-042affcf5015}, logIndex: 0, isSendImmediate: true 01:21:30,725 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:30,725 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:30,725 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:30,725 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:30,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:30,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): Number of entries to be appended = 1 01:21:30,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:21:30,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:30,726 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - new-server: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:30,726 PM [new-server-%d311905] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2B485CF0.tmp 01:21:30,726 PM [leader-%d863159] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2B36D259.v1 01:21:30,726 PM [leader-%d863159] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2B36D259.v1 01:21:30,726 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - new-server: Updated server config: isVoting: true, peers: [PeerInfo [id=leader, address=null, votingState=VOTING]] 01:21:30,726 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): Log size is now 1 01:21:30,726 PM [leader-%d863159] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2852E4ED.v1 01:21:30,727 PM [leader-%d863159] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/TermInfo.properties 01:21:30,727 PM [leader-%d863159] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/journal-v1-1.log 01:21:30,727 PM [leader-%d863159] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/metadata-v1-1.log 01:21:30,727 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=new-server, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=true, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:30,727 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached -1 (of expected -1) 01:21:30,727 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: awaiting more completions to resolve 1 synchronized callback(s) 01:21:30,727 PM [leader-%d863159] [DEBUG] RaftStorage - leader: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:30.676521197Z, file=/tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2852E4ED.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:30,727 PM [leader-%d863159] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:30.725013081Z, file=/tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2B36D259.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:30,727 PM [leader-%d863159] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:30,728 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server109, votingMember=true] 01:21:30,728 PM [new-server-%d311905] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2B485CF0.v1 01:21:30,728 PM [new-server-%d311905] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2B485CF0.v1 01:21:30,728 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:30,728 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:30,728 PM [new-server-%d311905] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2AEB2B62.v1 01:21:30,728 PM [new-server-%d311905] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/TermInfo.properties 01:21:30,729 PM [new-server-%d311905] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/journal-v1-1.log 01:21:30,729 PM [new-server-%d311905] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/metadata-v1-1.log 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for new-server updated: matchIndex: 0, nextIndex: 1 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:21:30,729 PM [new-server-%d311905] [DEBUG] RaftStorage - new-server: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:30.720055138Z, file=/tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2AEB2B62.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 0 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to 0 01:21:30,729 PM [new-server-%d311905] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:30.726162672Z, file=/tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2B485CF0.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:30,729 PM [new-server-%d311905] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:30,729 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:30,729 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - new-server: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server2, newServerAddress=pekko://test/user/follower107, votingMember=false] 01:21:30,729 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@15575a7d} 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: New follower is non-voting - directly persisting new server configuration 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]] 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Persist data index=1 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:30,729 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:30,730 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=fd323f83-f643-44f4-a82b-b70d0aa56199}, logIndex: 1, isSendImmediate: true 01:21:30,730 PM [leader-%d863160] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2B82E979.tmp 01:21:30,730 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:30,730 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 1 is present for follower new-server 01:21:30,730 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:30,730 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server2: active: false, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:30,730 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation AddServer [newServerId=new-server2, newServerAddress=pekko://test/user/follower107, votingMember=false] 01:21:30,730 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - new-server2: Last sync leader does not match current leader leader, need to catch up to -1 01:21:30,730 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server2 01:21:30,729 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: synchronized callbacks after 2.532 ms 01:21:30,730 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server2: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:30,730 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): Commit index set to 0 01:21:30,730 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:30,731 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 1 is present for follower new-server 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - new-server (Follower): Setting last applied to 0 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to 0 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - new-server: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): Number of entries to be appended = 1 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - new-server: Append log entry and persist JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - new-server: Updated server config: isVoting: true, peers: [PeerInfo [id=leader, address=null, votingState=VOTING], PeerInfo [id=new-server2, address=null, votingState=NON_VOTING]] 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): Log size is now 2 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=new-server, logLastIndex=1, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=true, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:30,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:30,731 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for new-server updated: matchIndex: 1, nextIndex: 2 01:21:30,731 PM [leader-%d863160] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2B82E979.v1 01:21:30,732 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:30,732 PM [new-server-%d311906] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2B9AD750.tmp 01:21:30,732 PM [leader-%d863160] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2B36D259.v1 01:21:30,732 PM [leader-%d863160] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2B82E979.v1 01:21:30,732 PM [leader-%d863160] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/TermInfo.properties 01:21:30,732 PM [leader-%d863160] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/journal-v1-1.log 01:21:30,732 PM [leader-%d863160] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5755105613448154436/leader/metadata-v1-1.log 01:21:30,733 PM [new-server-%d311906] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2B9AD750.v1 01:21:30,733 PM [new-server-%d311906] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2B485CF0.v1 01:21:30,734 PM [new-server-%d311906] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2B9AD750.v1 01:21:30,734 PM [new-server-%d311906] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/TermInfo.properties 01:21:30,734 PM [new-server-%d311906] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/journal-v1-1.log 01:21:30,734 PM [new-server-%d311906] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-5755105613448154436/new-server/metadata-v1-1.log 01:21:30,734 PM [new-server-%d311906] [DEBUG] RaftStorage - new-server: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:30.726162672Z, file=/tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2B485CF0.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:30,734 PM [new-server-%d311906] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:30.731567952Z, file=/tmp/junit-5755105613448154436/new-server/snapshot-0000000068B9925A-2B9AD750.v1, size=100, entryCompress=NONE, stateCompress=NONE} 01:21:30,734 PM [new-server-%d311906] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:30,737 PM [leader-%d863160] [DEBUG] RaftStorage - leader: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:30.725013081Z, file=/tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2B36D259.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:30,737 PM [leader-%d863160] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:30.729999737Z, file=/tmp/junit-5755105613448154436/leader/snapshot-0000000068B9925A-2B82E979.v1, size=100, entryCompress=NONE, stateCompress=NONE} 01:21:30,737 PM [leader-%d863160] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:30,737 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:30,737 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:30,738 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 01:21:30,738 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 1 01:21:30,738 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from 0 to 1 01:21:30,738 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:30,739 PM [test-pekko.actor.default-dispatcher-11] [ERROR] OneForOneStrategy - Cannot invoke "org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.getReplicatedToAllIndex()" because "currentBehavior" is null java.lang.NullPointerException: Cannot invoke "org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.getReplicatedToAllIndex()" because "currentBehavior" is null at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.SnapshotManager.trimLog(SnapshotManager.java:663) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.performSnapshotWithoutCapture(RaftActorBehavior.java:438) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.Follower.handleAppendEntries(Follower.java:222) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.appendEntries(RaftActorBehavior.java:152) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.handleMessage(RaftActorBehavior.java:375) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.Follower.handleMessage(Follower.java:502) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.ForwardMessageToBehaviorActor.onReceive(ForwardMessageToBehaviorActor.java:25) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.UntypedAbstractActor$$anon$1.applyOrElse(AbstractActor.scala:341) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:481) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:272) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:233) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:245) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188) [ERROR] [09/04/2025 13:21:30.738] [test-pekko.actor.internal-dispatcher-10] [pekko://test/user/follower107] Cannot invoke "org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.getReplicatedToAllIndex()" because "currentBehavior" is null java.lang.NullPointerException: Cannot invoke "org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.getReplicatedToAllIndex()" because "currentBehavior" is null at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.SnapshotManager.trimLog(SnapshotManager.java:663) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.performSnapshotWithoutCapture(RaftActorBehavior.java:438) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.Follower.handleAppendEntries(Follower.java:222) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.appendEntries(RaftActorBehavior.java:152) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.handleMessage(RaftActorBehavior.java:375) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.Follower.handleMessage(Follower.java:502) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.ForwardMessageToBehaviorActor.onReceive(ForwardMessageToBehaviorActor.java:25) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.UntypedAbstractActor$$anon$1.applyOrElse(AbstractActor.scala:341) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:481) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:272) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:233) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:245) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188) 01:21:30,741 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:30,741 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - new-server: persist complete JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:30,741 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@75bbfa58} 01:21:30,741 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: synchronized callbacks after 92.38 μs [WARN] [09/04/2025 13:21:30.741] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/new-server109] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=true] [INFO] [pekkoDeadLetter][09/04/2025 13:21:30.741] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/new-server109] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/new-server109] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:30,741 PM [test-pekko.actor.default-dispatcher-12] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/new-server109] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:30,738 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:30,903 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server2 01:21:30,903 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server2: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:30,903 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - new-server (Follower): Commit index set to 1 01:21:30,903 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 1 (of expected -1) 01:21:30,903 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - new-server (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 01:21:30,903 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - new-server (Follower): Setting last applied to 1 01:21:30,903 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from 0 to 1 01:21:30,903 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - new-server: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:30,932 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithOperationInProgress ending 01:21:30,933 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower107] 01:21:30,945 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector108#680289488] 01:21:30,945 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server109] 01:21:30,945 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#256,new-server-writer-35]/waiting 01:21:30,946 PM [new-server-writer-35] [DEBUG] JournalWriteTask - new-server: received 1 commands after 347.2 ms 01:21:30,946 PM [new-server-writer-35] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 40.16 μs 01:21:30,946 PM [new-server-writer-35] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:30,947 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:30,947 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:30,947 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:30,947 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:30,947 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader110] 01:21:30,947 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#259,leader-writer-36]/waiting 01:21:30,947 PM [leader-writer-36] [DEBUG] JournalWriteTask - leader: received 1 commands after 316.9 ms 01:21:30,947 PM [leader-writer-36] [DEBUG] JournalWriteTask - leader: completed 1 commands in 20.29 μs 01:21:30,947 PM [leader-writer-36] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:30,948 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:30,948 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - leader: journal closed 01:21:30,948 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:30,948 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:30,948 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leaderCollector111] 01:21:30,954 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServerNonExistentServer starting 01:21:30,955 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader113] [WARN] [09/04/2025 13:21:30.955] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader113]: org.opendaylight.controller.cluster.common.actor.Monitor@595a6c8a 01:21:30,955 PM [test-pekko.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader113] to Actor[pekko://test/user/termination-monitor] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [INFO] [pekkoDeadLetter][09/04/2025 13:21:30.955] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader113] to Actor[pekko://test/user/termination-monitor] was not delivered. [5] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:30,956 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:30,956 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:30,957 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: started executor 01:21:30,957 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: started executor 01:21:30,958 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:30,959 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:30,959 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:30,959 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:30,959 PM [leader-writer-37] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:30,959 PM [leader-writer-37] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:30,959 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/junit-3446655245126348376/leader/TermInfo.properties does not exist 01:21:30,962 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:30,962 PM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:30,962 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-3446655245126348376/leader/TermInfo.properties 01:21:30,962 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-3446655245126348376/leader/journal-v1-1.log 01:21:30,962 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-3446655245126348376/leader/metadata-v1-1.log 01:21:30,962 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: no eligible files found 01:21:30,962 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:30,962 PM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - leader: Recovery completed in in 3.188 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:30,962 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:30,963 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower] 01:21:30,963 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:21:30,974 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: onRemoveServer: RemoveServer{serverId='new-server'}, state: Idle 01:21:30,976 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServerNonExistentServer ending 01:21:30,976 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower112] 01:21:30,987 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader113] 01:21:30,988 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#269,leader-writer-37]/waiting 01:21:30,988 PM [leader-writer-37] [DEBUG] JournalWriteTask - leader: received 1 commands after 29.06 ms 01:21:30,988 PM [leader-writer-37] [DEBUG] JournalWriteTask - leader: completed 1 commands in 38.26 μs 01:21:30,988 PM [leader-writer-37] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:30,989 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:30,989 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - leader: journal closed 01:21:30,989 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:30,989 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:30,993 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServerWithNoLeader starting 01:21:30,993 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] MockRaftActor - Actor created TestActor[pekko://test/user/leader115] 01:21:30,994 PM [test-pekko.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader115] to Actor[pekko://test/user/termination-monitor] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. [WARN] [09/04/2025 13:21:30.994] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader115]: org.opendaylight.controller.cluster.common.actor.Monitor@2679f163 [INFO] [pekkoDeadLetter][09/04/2025 13:21:30.994] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader115] to Actor[pekko://test/user/termination-monitor] was not delivered. [6] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:30,995 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:30,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: started executor 01:21:30,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: started executor 01:21:30,996 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:30,997 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:30,997 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:30,997 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:30,997 PM [leader-writer-38] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:30,997 PM [leader-writer-38] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:30,997 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/junit-12011712402740293194/leader/TermInfo.properties does not exist 01:21:31,0 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=0, votedFor=null 01:21:31,0 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=0} 01:21:31,0 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12011712402740293194/leader/TermInfo.properties 01:21:31,0 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12011712402740293194/leader/journal-v1-1.log 01:21:31,0 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12011712402740293194/leader/metadata-v1-1.log 01:21:31,0 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: no eligible files found 01:21:31,0 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:31,0 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - leader: Recovery completed in in 3.077 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:31,0 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:31,11 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: onRemoveServer: RemoveServer{serverId='follower'}, state: Idle 01:21:31,12 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: No leader - returning NO_LEADER reply 01:21:31,12 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServerWithNoLeader ending 01:21:31,12 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower114] 01:21:31,23 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader115] 01:21:31,24 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#270,leader-writer-38]/waiting 01:21:31,24 PM [leader-writer-38] [DEBUG] JournalWriteTask - leader: received 1 commands after 26.67 ms 01:21:31,24 PM [leader-writer-38] [DEBUG] JournalWriteTask - leader: completed 1 commands in 31.32 μs 01:21:31,24 PM [leader-writer-38] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:31,24 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:31,24 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - leader: journal closed 01:21:31,24 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:31,24 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:31,29 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServerLeader starting 01:21:31,29 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader118] [WARN] [09/04/2025 13:21:31.030] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader118]: org.opendaylight.controller.cluster.common.actor.Monitor@9db0407 [INFO] [pekkoDeadLetter][09/04/2025 13:21:31.030] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader118] to Actor[pekko://test/user/termination-monitor] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:31,30 PM [test-pekko.actor.default-dispatcher-13] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/leader118] to Actor[pekko://test/user/termination-monitor] was not delivered. [7] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:31,31 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:31,31 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:31,31 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: started executor 01:21:31,31 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: started executor 01:21:31,32 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:31,33 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:31,33 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:31,33 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:31,33 PM [leader-writer-39] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:31,33 PM [leader-writer-39] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:31,33 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/junit-12500563010453326614/leader/TermInfo.properties does not exist 01:21:31,37 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:31,37 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:31,37 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12500563010453326614/leader/TermInfo.properties 01:21:31,37 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12500563010453326614/leader/journal-v1-1.log 01:21:31,37 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12500563010453326614/leader/metadata-v1-1.log 01:21:31,37 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - leader: no eligible files found 01:21:31,38 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:31,38 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - leader: Recovery completed in in 4.550 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:31,38 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:31,38 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower] 01:21:31,38 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [09/04/2025 13:21:31.038] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower117] received dead letter from TestActor[pekko://test/user/leader118]: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} [INFO] [pekkoDeadLetter][09/04/2025 13:21:31.038] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower117] Message [org.opendaylight.controller.cluster.raft.messages.AppendEntries] from TestActor[pekko://test/user/leader118] to Actor[pekko://test/user/follower117] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/follower117] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:31,39 PM [test-pekko.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.messages.AppendEntries] from TestActor[pekko://test/user/leader118] to Actor[pekko://test/user/follower117] was not delivered. [8] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/follower117] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:31,42 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/follower117] [WARN] [09/04/2025 13:21:31.042] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower117]: org.opendaylight.controller.cluster.common.actor.Monitor@4b383586 [INFO] [pekkoDeadLetter][09/04/2025 13:21:31.042] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower117] to Actor[pekko://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:31,42 PM [test-pekko.actor.default-dispatcher-14] [INFO] EmptyLocalActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.common.actor.Monitor] from TestActor[pekko://test/user/follower117] to Actor[pekko://test/user/termination-monitor] was not delivered. [9] dead letters encountered. If this is not an expected behavior then Actor[pekko://test/user/termination-monitor] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:31,43 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower: Starting recovery with journal batch size 1000 01:21:31,43 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: started executor 01:21:31,44 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: started executor 01:21:31,44 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower: metadata journal open: replayFrom=1 applyTo=0 01:21:31,44 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:31,44 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - follower: journal open: applyTo=0 01:21:31,45 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower: recovering to non-persistent 01:21:31,45 PM [follower-writer-40] [DEBUG] JournalWriteTask - follower: journal writer started 01:21:31,45 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/junit-12500563010453326614/follower/TermInfo.properties does not exist 01:21:31,45 PM [follower-writer-40] [DEBUG] JournalWriteTask - follower: waiting to receive commands 01:21:31,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=0, votedFor=null 01:21:31,47 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - follower: Local TermInfo store seeded with TermInfo{term=0} 01:21:31,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-12500563010453326614/follower/TermInfo.properties 01:21:31,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-12500563010453326614/follower/journal-v1-1.log 01:21:31,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-12500563010453326614/follower/metadata-v1-1.log 01:21:31,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: no eligible files found 01:21:31,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from -1 to -1 01:21:31,47 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - follower: Recovery completed in in 2.556 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:31,47 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower: Recovery completed and can restore from snapshot 01:21:31,58 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: onRemoveServer: RemoveServer{serverId='leader'}, state: Idle 01:21:31,61 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=follower, isVoting=true]] 01:21:31,61 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: Persist data index=0 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]} 01:21:31,61 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]}, pending=true} 01:21:31,61 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=b45e51a2-47fd-4eb0-8963-61570dac6486}, logIndex: 0, isSendImmediate: true 01:21:31,61 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:31,61 PM [leader-%d120630] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-12500563010453326614/leader/snapshot-0000000068B9925B-03ADE325.tmp 01:21:31,61 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:31,62 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:31,62 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation RemoveServer{serverId='leader'} 01:21:31,64 PM [leader-%d120630] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-12500563010453326614/leader/snapshot-0000000068B9925B-03ADE325.v1 01:21:31,64 PM [leader-%d120630] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-12500563010453326614/leader/snapshot-0000000068B9925B-03ADE325.v1 01:21:31,64 PM [leader-%d120630] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12500563010453326614/leader/TermInfo.properties 01:21:31,64 PM [leader-%d120630] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12500563010453326614/leader/journal-v1-1.log 01:21:31,64 PM [leader-%d120630] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12500563010453326614/leader/metadata-v1-1.log 01:21:31,64 PM [leader-%d120630] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:31.061727525Z, file=/tmp/junit-12500563010453326614/leader/snapshot-0000000068B9925B-03ADE325.v1, size=75, entryCompress=NONE, stateCompress=NONE} 01:21:31,65 PM [leader-%d120630] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:31,65 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:31,65 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]}, pending=true} 01:21:31,65 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=1, votedFor=null 01:21:31,65 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower: Last sync leader does not match current leader leader, need to catch up to -1 [WARN] [09/04/2025 13:21:31.065] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower117] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower, initialSyncDone=false] [INFO] [pekkoDeadLetter][09/04/2025 13:21:31.065] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower117] Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower117] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:31,65 PM [test-pekko.actor.default-dispatcher-14] [INFO] TestActorRef - pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.base.messages.FollowerInitialSyncUpStatus] to TestActor[pekko://test/user/follower117] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'. 01:21:31,66 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: -1, nextIndex: 0 01:21:31,66 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:31,66 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower follower 01:21:31,66 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower follower: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]}, pending=false}]} 01:21:31,66 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]}, pending=false}]} 01:21:31,66 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): Number of entries to be appended = 1 01:21:31,66 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:21:31,66 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]} 01:21:31,66 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]}, pending=false} 01:21:31,66 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower: Updated server config: isVoting: true, peers: [] 01:21:31,66 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): Log size is now 1 01:21:31,66 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=follower, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=true, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:31,66 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached -1 (of expected -1) 01:21:31,66 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower: awaiting more completions to resolve 1 synchronized callback(s) 01:21:31,66 PM [follower-%d536225] [DEBUG] RaftStorage - follower: starting snapshot writeout to /tmp/junit-12500563010453326614/follower/snapshot-0000000068B9925B-03F85C9C.tmp 01:21:31,67 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 0, nextIndex: 1 01:21:31,67 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:21:31,67 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 0 01:21:31,67 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to 0 01:21:31,67 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]} 01:21:31,67 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:31,67 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:31,68 PM [follower-%d536225] [DEBUG] RaftStorage - follower: finished snapshot writeout to /tmp/junit-12500563010453326614/follower/snapshot-0000000068B9925B-03F85C9C.v1 01:21:31,68 PM [follower-%d536225] [DEBUG] RaftStorage - follower: selected SNAPSHOT_V1 to handle file /tmp/junit-12500563010453326614/follower/snapshot-0000000068B9925B-03F85C9C.v1 01:21:31,68 PM [follower-%d536225] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-12500563010453326614/follower/TermInfo.properties 01:21:31,69 PM [follower-%d536225] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-12500563010453326614/follower/journal-v1-1.log 01:21:31,69 PM [follower-%d536225] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-12500563010453326614/follower/metadata-v1-1.log 01:21:31,69 PM [follower-%d536225] [DEBUG] RaftStorage - follower: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:31.066608284Z, file=/tmp/junit-12500563010453326614/follower/snapshot-0000000068B9925B-03F85C9C.v1, size=75, entryCompress=NONE, stateCompress=NONE} 01:21:31,69 PM [follower-%d536225] [DEBUG] RaftStorageCompleter - follower: 1 completion(s) pending 01:21:31,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower: running 1 completion(s) 01:21:31,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]}, pending=false} 01:21:31,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@115b48df} 01:21:31,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower: synchronized callbacks after 2.523 ms 01:21:31,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): Commit index set to 0 01:21:31,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:31,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:21:31,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower (Follower): Setting last applied to 0 01:21:31,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from -1 to 0 01:21:31,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true]]} [WARN] [09/04/2025 13:21:31.069] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower117] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower, initialSyncDone=true] [WARN] [09/04/2025 13:21:31.070] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/leader118] unhandled message from TestActor[pekko://test/user/leader118]: ServerRemoved [serverId=leader] 01:21:31,113 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServerLeader ending 01:21:31,113 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower116] 01:21:31,125 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader118] 01:21:31,125 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#271,leader-writer-39]/waiting 01:21:31,125 PM [leader-writer-39] [DEBUG] JournalWriteTask - leader: received 1 commands after 92.03 ms 01:21:31,125 PM [leader-writer-39] [DEBUG] JournalWriteTask - leader: completed 1 commands in 30.60 μs 01:21:31,125 PM [leader-writer-39] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:31,125 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:31,125 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - leader: journal closed 01:21:31,126 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:31,126 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:31,126 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leaderCollector119] 01:21:31,126 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector120#-938098252] 01:21:31,127 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower117] 01:21:31,127 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EnabledRaftStorage - follower: terminating thread VirtualThread[#272,follower-writer-40]/waiting 01:21:31,127 PM [follower-writer-40] [DEBUG] JournalWriteTask - follower: received 1 commands after 82.75 ms 01:21:31,127 PM [follower-writer-40] [DEBUG] JournalWriteTask - follower: completed 1 commands in 19.62 μs 01:21:31,128 PM [follower-writer-40] [DEBUG] JournalWriteTask - follower: journal writer stopped 01:21:31,128 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - follower: journal closed 01:21:31,128 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - follower: journal closed 01:21:31,128 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:31,128 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:31,136 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeToVotingWithNoLeaderAndOtherLeaderElected starting 01:21:31,138 PM [main] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=1, votedFor=node1 01:21:31,139 PM [main] [DEBUG] EntryJournalV1 - node1: metadata journal open: replayFrom=1 applyTo=0 01:21:31,139 PM [main] [DEBUG] EntryJournalV1 - node1: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:31,143 PM [main] [DEBUG] EntryJournalV1 - node1: journal closed 01:21:31,145 PM [main] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=1, votedFor=node1 01:21:31,145 PM [main] [DEBUG] EntryJournalV1 - node2: metadata journal open: replayFrom=1 applyTo=0 01:21:31,146 PM [main] [DEBUG] EntryJournalV1 - node2: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:31,149 PM [main] [DEBUG] EntryJournalV1 - node2: journal closed 01:21:31,150 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/node1] [WARN] [09/04/2025 13:21:31.150] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/node1]: org.opendaylight.controller.cluster.common.actor.Monitor@b9bcb45 01:21:31,151 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - node1: Starting recovery with journal batch size 1000 01:21:31,152 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node1: started executor 01:21:31,152 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node1: started executor 01:21:31,152 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - node1: metadata journal open: replayFrom=1 applyTo=0 01:21:31,153 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - node1: journal open: firstIndex=1 lastIndex=1 replayFrom=1 fileEntries=0 01:21:31,153 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - node1: journal open: applyTo=0 01:21:31,153 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: journal writer started 01:21:31,153 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: waiting to receive commands 01:21:31,153 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - node1: starting journal recovery 01:21:31,153 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=1, votedFor=node1 01:21:31,154 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-10981657468854361894/node1/TermInfo.properties 01:21:31,154 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-10981657468854361894/node1/journal-v1-1.log 01:21:31,154 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-10981657468854361894/node1/metadata-v1-1.log 01:21:31,154 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node1: no eligible files found 01:21:31,154 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from -1 to -1 01:21:31,154 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - node1: applying entries up to 0 01:21:31,154 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - node1: recovered journal DefaultLogEntry[index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=true]]}] 01:21:31,155 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - node1: Updated server config: isVoting: false, peers: [PeerInfo [id=node2, address=null, votingState=VOTING]] 01:21:31,155 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from -1 to -1 01:21:31,155 PM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - node1: Recovery completed in in 1.434 ms: last log index = 0, last log term = 1, napshot index = -1, snapshot term = -1, journal size = 1 01:21:31,155 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - node1: Recovery completed and cannot restore from snapshot 01:21:31,157 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/node2] [WARN] [09/04/2025 13:21:31.157] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/node2]: org.opendaylight.controller.cluster.common.actor.Monitor@ae42a81 01:21:31,157 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - node2: Starting recovery with journal batch size 1000 01:21:31,158 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node2: started executor 01:21:31,158 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node2: started executor 01:21:31,158 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - node2: metadata journal open: replayFrom=1 applyTo=0 01:21:31,158 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - node2: journal open: firstIndex=1 lastIndex=1 replayFrom=1 fileEntries=0 01:21:31,158 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - node2: journal open: applyTo=0 01:21:31,158 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: journal writer started 01:21:31,158 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: waiting to receive commands 01:21:31,158 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - node2: starting journal recovery 01:21:31,158 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=1, votedFor=node1 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-10981657468854361894/node2/TermInfo.properties 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-10981657468854361894/node2/journal-v1-1.log 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-10981657468854361894/node2/metadata-v1-1.log 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - node2: no eligible files found 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from -1 to -1 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - node2: applying entries up to 0 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] JournalRecovery - node2: recovered journal DefaultLogEntry[index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=true]]}] 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - node2: Updated server config: isVoting: true, peers: [PeerInfo [id=node1, address=null, votingState=NON_VOTING]] 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from -1 to -1 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - node2: Recovery completed in in 655.1 μs: last log index = 0, last log term = 1, napshot index = -1, snapshot term = -1, journal size = 1 01:21:31,159 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - node2: Recovery completed and cannot restore from snapshot 01:21:31,170 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - node1: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={node1=true, node2=true}, serversVisited=[]], state: Idle 01:21:31,173 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:31,173 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - node1: Sending local ElectionTimeout to start leader election 01:21:31,173 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - node1: Updated server config: isVoting: true, peers: [PeerInfo [id=node2, address=null, votingState=VOTING]] 01:21:31,174 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:31,174 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - node1 (Follower): Received TimeoutNow - switching to Candidate 01:21:31,174 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - node1 (Candidate): Election: Candidate has following voting peers: [node2] 01:21:31,176 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=2, votedFor=node1 01:21:31,176 PM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - node1 (Candidate): Starting new election term 2 01:21:31,176 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - node1 (Candidate): Sending RequestVote{term=2, candidateId=node1, lastLogIndex=0, lastLogTerm=1} to peer node2 01:21:31,176 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - node1 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:21:31,221 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node2 (Follower): Received TimeoutNow - switching to Candidate 01:21:31,221 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - node2 (Candidate): Election: Candidate has following voting peers: [] 01:21:31,223 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=2, votedFor=node2 01:21:31,223 PM [test-pekko.actor.default-dispatcher-11] [INFO] Candidate - node2 (Candidate): Starting new election term 2 01:21:31,223 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - node2 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:21:31,223 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Candidate - node2 (Candidate): Received ElectionTimeout 01:21:31,224 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - node2 (Leader): Election: Leader has following peers: [node1] 01:21:31,224 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - node2 (Leader): Checking sendAppendEntries for follower node1: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:31,224 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorBehavior - node2 (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:21:31,224 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - node1 (Candidate): RaftRPC message received AppendEntries{term=2, leaderId=node2, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}, my term is 2 01:21:31,224 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - node1 (Candidate): handleAppendEntries: AppendEntries{term=2, leaderId=node2, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 01:21:31,224 PM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - node1 (Candidate): New Leader node2 sent an AppendEntries to Candidate for term 2 - will switch to Follower 01:21:31,224 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - node1 (Candidate) :- Switching from behavior Candidate to Follower, election term: 2 01:21:31,342 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - node1: Last sync leader does not match current leader node2, need to catch up to -1 01:21:31,342 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): handleAppendEntriesReply - FollowerLogInformation for node1 updated: matchIndex: 0, nextIndex: 1 01:21:31,342 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - node1: New leader node2 elected 01:21:31,343 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - node1: Forwarding ChangeServersVotingStatus [serverVotingStatusMap={node1=true, node2=true}, serversVisited=[]] to new leader 01:21:31,343 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Not updating commit index to 0 - retrieved log entry with index 0, term 1 does not match the current term 2 01:21:31,343 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - node1: Updated server config: isVoting: false, peers: [PeerInfo [id=node2, address=pekko://test/user/node2, votingState=VOTING]] 01:21:31,343 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - node1: getLeaderAddress leaderId = node2 peerAddress = pekko://test/user/node2 01:21:31,343 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Checking sendAppendEntries for follower node1: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:31,343 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - node1: Not leader - forwarding to leader ActorSelection[Anchor(pekko://test/), Path(/user/node2)] [WARN] [09/04/2025 13:21:31.343] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/node1] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=node1, initialSyncDone=false] 01:21:31,343 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - node2: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={node1=true, node2=true}, serversVisited=[]], state: Idle 01:21:31,343 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:31,343 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - node2: Updated server config: isVoting: true, peers: [PeerInfo [id=node1, address=pekko://test/user/node1, votingState=VOTING]] 01:21:31,344 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - node2: New server configuration : [ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]] 01:21:31,344 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - node2: Persist data index=1 term=2 command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]} 01:21:31,344 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - node2: Append log entry and persist JournaledLogEntry{index=1, term=2, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=true} 01:21:31,344 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=4f66fc05-b54e-4c02-97a6-227dbb291ea1}, logIndex: 1, isSendImmediate: true 01:21:31,344 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Checking sendAppendEntries for follower node1: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:31,344 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): sendAppendEntries: 1 is present for follower node1 01:21:31,344 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: received 1 commands after 185.7 ms 01:21:31,344 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Sending AppendEntries to follower node1: AppendEntries{term=2, leaderId=node2, prevLogIndex=0, prevLogTerm=1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=true}]} 01:21:31,344 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node1 (Follower): handleAppendEntries: AppendEntries{term=2, leaderId=node2, prevLogIndex=0, prevLogTerm=1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=2, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=true}]} 01:21:31,344 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node1 (Follower): Number of entries to be appended = 1 01:21:31,344 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node1 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 01:21:31,345 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node1 (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]} 01:21:31,345 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - node1: Append log entry and persist JournaledLogEntry{index=1, term=2, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=true} 01:21:31,345 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - node1: Updated server config: isVoting: true, peers: [PeerInfo [id=node2, address=pekko://test/user/node2, votingState=VOTING]] 01:21:31,345 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node1 (Follower): Log size is now 2 01:21:31,345 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node1 (Follower): handleAppendEntries returning : AppendEntriesReply{term=2, success=true, followerId=node1, logLastIndex=1, logLastTerm=2, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:31,345 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - node1: Lagging 0 entries behind leader node2 and reached -1 (of expected -1) 01:21:31,345 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - node1: awaiting more completions to resolve 1 synchronized callback(s) 01:21:31,345 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: received 1 commands after 191.6 ms 01:21:31,345 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - node2: Returning OK for operation ChangeServersVotingStatus [serverVotingStatusMap={node1=true, node2=true}, serversVisited=[]] 01:21:31,345 PM [node2-writer-42] [DEBUG] RaftStorageCompleter - node2: 1 completion(s) pending 01:21:31,345 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: completed 1 commands in 1.277 ms 01:21:31,346 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: waiting to receive commands 01:21:31,346 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - node2: running 1 completion(s) 01:21:31,346 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - node2: persist complete JournaledLogEntry{index=1, term=2, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=true} 01:21:31,346 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Not updating commit index to 0 - retrieved log entry with index 0, term 1 does not match the current term 2 01:21:31,346 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): handleAppendEntriesReply - FollowerLogInformation for node1 updated: matchIndex: 1, nextIndex: 2 01:21:31,346 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Not updating commit index to 0 - retrieved log entry with index 0, term 1 does not match the current term 2 01:21:31,346 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: -1 01:21:31,346 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - node2 (Leader): Setting last applied to 0 01:21:31,346 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from -1 to 0 01:21:31,346 PM [node1-writer-41] [DEBUG] RaftStorageCompleter - node1: 1 completion(s) pending 01:21:31,346 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - node2: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=true]]} 01:21:31,346 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - node1: running 1 completion(s) 01:21:31,346 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - node1: persist complete JournaledLogEntry{index=1, term=2, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=false} 01:21:31,346 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: completed 1 commands in 1.204 ms 01:21:31,346 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: waiting to receive commands 01:21:31,346 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - node1: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=2, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@1a33597c} 01:21:31,346 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - node1: synchronized callbacks after 1.416 ms [WARN] [09/04/2025 13:21:31.346] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/node1] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=node1, initialSyncDone=true] 01:21:31,347 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - node2 (Leader): Setting last applied to 1 01:21:31,347 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from 0 to 1 01:21:31,347 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - node2: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]} 01:21:31,347 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - node2: fakeSnapshot purging log to 0 for term 2 01:21:31,347 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: received 1 commands after 1.235 ms 01:21:31,347 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Checking sendAppendEntries for follower node1: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 01:21:31,347 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node1 (Follower): Commit index set to 1 01:21:31,347 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - node1: Lagging 0 entries behind leader node2 and reached 1 (of expected -1) 01:21:31,347 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node1 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 1 01:21:31,347 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - node1 (Follower): Setting last applied to 0 01:21:31,347 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from -1 to 0 01:21:31,347 PM [node2-writer-42] [DEBUG] EntryJournalV1 - node2: updated meta to JournalMeta[replayFrom=1, applyTo=2] 01:21:31,347 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - node1: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=true]]} 01:21:31,347 PM [node2-writer-42] [DEBUG] RaftStorageCompleter - node2: 1 completion(s) pending 01:21:31,347 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - node1 (Follower): Setting last applied to 1 01:21:31,347 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from 0 to 1 01:21:31,347 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - node1: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]} 01:21:31,347 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: completed 1 commands in 322.3 μs 01:21:31,347 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: waiting to receive commands 01:21:31,347 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - node1: fakeSnapshot purging log to 0 for term 2 01:21:31,347 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: received 1 commands after 1.135 ms 01:21:31,347 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - node2: running 1 completion(s) 01:21:31,348 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorVotingConfigSupport - node2: VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:31,348 PM [node1-writer-41] [DEBUG] EntryJournalV1 - node1: updated meta to JournalMeta[replayFrom=1, applyTo=2] 01:21:31,348 PM [node1-writer-41] [DEBUG] RaftStorageCompleter - node1: 1 completion(s) pending 01:21:31,348 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: completed 1 commands in 325.8 μs 01:21:31,348 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: waiting to receive commands 01:21:31,348 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - node1: running 1 completion(s) 01:21:31,463 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - node1: Lagging 0 entries behind leader node2 and reached 1 (of expected -1) 01:21:31,549 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeToVotingWithNoLeaderAndOtherLeaderElected ending 01:21:31,549 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower122] 01:21:31,561 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector123#-1911986532] 01:21:31,561 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/node1] 01:21:31,561 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EnabledRaftStorage - node1: terminating thread VirtualThread[#275,node1-writer-41]/waiting 01:21:31,561 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: received 1 commands after 213.5 ms 01:21:31,562 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: completed 1 commands in 22.88 μs 01:21:31,562 PM [node1-writer-41] [DEBUG] JournalWriteTask - node1: journal writer stopped 01:21:31,562 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - node1: journal closed 01:21:31,562 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - node1: journal closed 01:21:31,562 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - node1: stopped executor with 0 remaining tasks 01:21:31,562 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - node1: stopped executor with 0 remaining tasks 01:21:31,562 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector124#246560876] 01:21:31,563 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/node2] 01:21:31,563 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EnabledRaftStorage - node2: terminating thread VirtualThread[#276,node2-writer-42]/waiting 01:21:31,563 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: received 1 commands after 215.9 ms 01:21:31,563 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: completed 1 commands in 19.00 μs 01:21:31,563 PM [node2-writer-42] [DEBUG] JournalWriteTask - node2: journal writer stopped 01:21:31,564 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - node2: journal closed 01:21:31,564 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - node2: journal closed 01:21:31,564 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - node2: stopped executor with 0 remaining tasks 01:21:31,564 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - node2: stopped executor with 0 remaining tasks 01:21:31,569 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithInstallSnapshotTimeout starting 01:21:31,570 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server127] [WARN] [09/04/2025 13:21:31.571] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server127]: org.opendaylight.controller.cluster.common.actor.Monitor@1fc204c5 01:21:31,572 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:31,572 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - new-server: started executor 01:21:31,572 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - new-server: started executor 01:21:31,573 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:31,573 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:31,573 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:31,573 PM [new-server-writer-43] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:31,574 PM [new-server-writer-43] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:31,574 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:31,574 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/junit-9318680855472017196/new-server/TermInfo.properties does not exist 01:21:31,576 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:31,576 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:31,576 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-9318680855472017196/new-server/TermInfo.properties 01:21:31,576 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-9318680855472017196/new-server/journal-v1-1.log 01:21:31,576 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-9318680855472017196/new-server/metadata-v1-1.log 01:21:31,577 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:31,577 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:31,577 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - new-server: Recovery completed in in 2.971 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:31,577 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:31,577 PM [test-pekko.actor.default-dispatcher-11] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:31,578 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader128] [WARN] [09/04/2025 13:21:31.578] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader128]: org.opendaylight.controller.cluster.common.actor.Monitor@ff17094 01:21:31,579 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:31,579 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:31,579 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: started executor 01:21:31,579 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: started executor 01:21:31,580 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:31,580 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:31,580 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:31,581 PM [leader-writer-44] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:31,581 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:31,581 PM [leader-writer-44] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:31,581 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/junit-9318680855472017196/leader/TermInfo.properties does not exist 01:21:31,584 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:31,584 PM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:31,585 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-9318680855472017196/leader/TermInfo.properties 01:21:31,585 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-9318680855472017196/leader/journal-v1-1.log 01:21:31,585 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-9318680855472017196/leader/metadata-v1-1.log 01:21:31,585 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: no eligible files found 01:21:31,585 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:31,585 PM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - leader: Recovery completed in in 4.091 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:31,585 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:31,585 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [] 01:21:31,596 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server127, votingMember=true], state: Idle 01:21:31,596 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server127, votingMember=true] 01:21:31,596 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 01:21:31,596 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on new-server 01:21:31,596 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - leader: last applied journal index prior to capture: 0 01:21:31,596 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader: support() called 01:21:31,597 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating capture snapshot for new server new-server 01:21:31,597 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: onUnInitializedFollowerSnapshotReply: UnInitializedFollowerSnapshotReply [followerId=bogus] 01:21:31,597 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Dropping UnInitializedFollowerSnapshotReply for server bogus: server Id doesn't match 01:21:31,597 PM [leader-%d840394] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:31,597 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:31,597 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader: Persising snapshot at EntryInfo[index=-1, term=-1]/EntryInfo[index=-1, term=-1] 01:21:31,597 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader: support() called 01:21:31,598 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 01:21:31,598 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): sendInstallSnapshot 01:21:31,598 PM [leader-%d840395] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-9318680855472017196/leader/snapshot-0000000068B9925B-23A6BD1C.tmp 01:21:31,598 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Snapshot 157 bytes, total chunks to send: 1 01:21:31,598 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Next chunk: total length=157, offset=0, size=157, hashCode=1546036366 01:21:31,598 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): next snapshot chunk size for follower new-server: 157 01:21:31,598 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): InstallSnapshot sent to follower Vector(user, new-server127), Chunk: 1/1 01:21:31,600 PM [leader-%d840395] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-9318680855472017196/leader/snapshot-0000000068B9925B-23A6BD1C.v1 01:21:31,600 PM [leader-%d840395] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-9318680855472017196/leader/snapshot-0000000068B9925B-23A6BD1C.v1 01:21:31,600 PM [leader-%d840395] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-9318680855472017196/leader/TermInfo.properties 01:21:31,600 PM [leader-%d840395] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-9318680855472017196/leader/journal-v1-1.log 01:21:31,600 PM [leader-%d840395] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-9318680855472017196/leader/metadata-v1-1.log 01:21:31,600 PM [leader-%d840395] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:31.598129948Z, file=/tmp/junit-9318680855472017196/leader/snapshot-0000000068B9925B-23A6BD1C.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:31,600 PM [leader-%d840395] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:31,600 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:31,600 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader: snapshot is durable as of 2025-09-04T13:21:31.598129948Z 01:21:31,601 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - leader: Snapshot success, discarding journal entries up to 0 01:21:31,601 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorSnapshotMessageSupport - leader: SnapshotComplete received 01:21:31,713 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - new-server (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:31,716 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=1, votedFor=null 01:21:31,716 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - new-server: Last sync leader does not match current leader leader, need to catch up to -1 [WARN] [09/04/2025 13:21:31.716] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/new-server127] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=false] 01:21:31,716 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for new-server updated: matchIndex: -1, nextIndex: 0 01:21:31,812 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: handleInstallSnapshotTimeout for new server new-server 01:21:31,812 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Returning TIMEOUT for operation AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server127, votingMember=true] 01:21:31,813 PM [test-pekko.actor.default-dispatcher-12] [WARN] RaftActorVotingConfigSupport - leader: Timeout occured for new server new-server while installing snapshot 01:21:31,813 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithInstallSnapshotTimeout ending 01:21:31,813 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower125] 01:21:31,815 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector126#-1135032039] 01:21:31,815 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server127] 01:21:31,815 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#279,new-server-writer-43]/waiting 01:21:31,816 PM [new-server-writer-43] [DEBUG] JournalWriteTask - new-server: received 1 commands after 242.2 ms 01:21:31,816 PM [new-server-writer-43] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 25.26 μs 01:21:31,816 PM [new-server-writer-43] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:31,816 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:31,816 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:31,816 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:31,816 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:31,817 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader128] 01:21:31,817 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#280,leader-writer-44]/waiting 01:21:31,817 PM [leader-writer-44] [DEBUG] JournalWriteTask - leader: received 1 commands after 236.6 ms 01:21:31,817 PM [leader-writer-44] [DEBUG] JournalWriteTask - leader: completed 1 commands in 20.66 μs 01:21:31,817 PM [leader-writer-44] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:31,818 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:31,818 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - leader: journal closed 01:21:31,818 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:31,818 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:31,822 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithNoExistingFollower starting 01:21:31,826 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server131] [WARN] [09/04/2025 13:21:31.826] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server131]: org.opendaylight.controller.cluster.common.actor.Monitor@64cb8b1a 01:21:31,828 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:31,828 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - new-server: started executor 01:21:31,828 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - new-server: started executor 01:21:31,829 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:31,829 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:31,829 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:31,829 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:31,829 PM [new-server-writer-45] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:31,829 PM [new-server-writer-45] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:31,830 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/junit-7448345448504681977/new-server/TermInfo.properties does not exist 01:21:31,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:31,832 PM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:31,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7448345448504681977/new-server/TermInfo.properties 01:21:31,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7448345448504681977/new-server/journal-v1-1.log 01:21:31,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7448345448504681977/new-server/metadata-v1-1.log 01:21:31,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:31,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:31,832 PM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - new-server: Recovery completed in in 2.691 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:31,832 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:31,833 PM [test-pekko.actor.default-dispatcher-5] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:31,833 PM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1 01:21:31,834 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader132] [WARN] [09/04/2025 13:21:31.834] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader132]: org.opendaylight.controller.cluster.common.actor.Monitor@1b30b854 01:21:31,834 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:31,834 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:31,834 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: started executor 01:21:31,834 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: started executor 01:21:31,835 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:31,835 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:31,835 PM [test-pekko.actor.default-dispatcher-5] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:31,835 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:31,835 PM [leader-writer-46] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:31,835 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - /tmp/junit-7448345448504681977/leader/TermInfo.properties does not exist 01:21:31,835 PM [leader-writer-46] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:31,837 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:31,837 PM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:31,838 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7448345448504681977/leader/TermInfo.properties 01:21:31,838 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7448345448504681977/leader/journal-v1-1.log 01:21:31,838 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7448345448504681977/leader/metadata-v1-1.log 01:21:31,838 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorage - leader: no eligible files found 01:21:31,838 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:31,838 PM [test-pekko.actor.default-dispatcher-5] [INFO] Recovery - leader: Recovery completed in in 2.406 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:31,838 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:31,838 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [] 01:21:31,841 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server131, votingMember=true], state: Idle 01:21:31,841 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server131, votingMember=true] 01:21:31,841 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=1, lastAppliedTerm=1, lastIndex=1, lastTerm=1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on new-server 01:21:31,841 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - leader: last applied journal index prior to capture: 2 01:21:31,841 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader: support() called 01:21:31,841 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating capture snapshot for new server new-server 01:21:31,842 PM [leader-%d23317] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:31,842 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:31,842 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader: Persising snapshot at EntryInfo[index=1, term=1]/EntryInfo[index=1, term=1] 01:21:31,842 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - leader: support() called 01:21:31,842 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 01:21:31,842 PM [leader-%d23318] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-7448345448504681977/leader/snapshot-0000000068B9925B-323890A7.tmp 01:21:31,842 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): sendInstallSnapshot 01:21:31,842 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Snapshot 157 bytes, total chunks to send: 1 01:21:31,842 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Next chunk: total length=157, offset=0, size=157, hashCode=1546036366 01:21:31,842 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): next snapshot chunk size for follower new-server: 157 01:21:31,842 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): InstallSnapshot sent to follower Vector(user, new-server131), Chunk: 1/1 01:21:31,843 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - new-server (Follower): Term 1 in "InstallSnapshot{term=1, leaderId=leader, lastIncludedIndex=1, lastIncludedTerm=1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1]}" message is greater than follower's term 0 - updating term 01:21:31,845 PM [leader-%d23318] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-7448345448504681977/leader/snapshot-0000000068B9925B-323890A7.v1 01:21:31,845 PM [leader-%d23318] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-7448345448504681977/leader/snapshot-0000000068B9925B-323890A7.v1 01:21:31,845 PM [leader-%d23318] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7448345448504681977/leader/TermInfo.properties 01:21:31,845 PM [leader-%d23318] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7448345448504681977/leader/journal-v1-1.log 01:21:31,845 PM [leader-%d23318] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7448345448504681977/leader/metadata-v1-1.log 01:21:31,845 PM [leader-%d23318] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:31.842567847Z, file=/tmp/junit-7448345448504681977/leader/snapshot-0000000068B9925B-323890A7.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:31,845 PM [leader-%d23318] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:31,845 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:31,845 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - leader: snapshot is durable as of 2025-09-04T13:21:31.842567847Z 01:21:31,845 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - leader: Snapshot success, discarding journal entries up to 2 01:21:31,845 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorSnapshotMessageSupport - leader: SnapshotComplete received 01:21:31,845 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=1, votedFor=null 01:21:31,845 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleInstallSnapshot: InstallSnapshot{term=1, leaderId=leader, lastIncludedIndex=1, lastIncludedTerm=1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1]} 01:21:31,846 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - new-server: Last sync leader does not match current leader leader, need to catch up to 1 01:21:31,846 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotTracker - new-server (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 01:21:31,846 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - new-server (Follower): Snapshot received from leader: leader 01:21:31,846 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - new-server: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=157}} 01:21:31,846 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - new-server: support() called [WARN] [09/04/2025 13:21:31.846] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/new-server131] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=false] 01:21:31,847 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - new-server: Converted InstallSnapshot from leader: leader to state 01:21:31,847 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - new-server: last applied journal index prior to persisting applied snapshot: 0 01:21:31,847 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - new-server: support() called 01:21:31,847 PM [new-server-%d689515] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-7448345448504681977/new-server/snapshot-0000000068B9925B-3284E9A8.tmp 01:21:31,849 PM [new-server-%d689515] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-7448345448504681977/new-server/snapshot-0000000068B9925B-3284E9A8.v1 01:21:31,849 PM [new-server-%d689515] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-7448345448504681977/new-server/snapshot-0000000068B9925B-3284E9A8.v1 01:21:31,849 PM [new-server-%d689515] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7448345448504681977/new-server/TermInfo.properties 01:21:31,849 PM [new-server-%d689515] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7448345448504681977/new-server/journal-v1-1.log 01:21:31,849 PM [new-server-%d689515] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7448345448504681977/new-server/metadata-v1-1.log 01:21:31,849 PM [new-server-%d689515] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:31.847571368Z, file=/tmp/junit-7448345448504681977/new-server/snapshot-0000000068B9925B-3284E9A8.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:31,849 PM [new-server-%d689515] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:31,850 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:31,850 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - new-server: snapshot is durable as of 2025-09-04T13:21:31.847571368Z 01:21:31,850 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - new-server: support() called 01:21:31,850 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - new-server: applySnapshot called 01:21:31,850 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=1, followerId=new-server, chunkIndex=1, success=true} 01:21:31,850 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - new-server: Snapshot success, discarding journal entries up to 0 01:21:31,850 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorSnapshotMessageSupport - new-server: SnapshotComplete received 01:21:31,850 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=1, followerId=new-server, chunkIndex=1, success=true} 01:21:31,850 PM [test-pekko.actor.default-dispatcher-5] [INFO] AbstractLeader - leader (Leader): Snapshot successfully installed on follower new-server (last chunk 1) - matchIndex set to 1, nextIndex set to 2 01:21:31,850 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - Sent message UnInitializedFollowerSnapshotReply to self 01:21:31,850 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - leader: fakeSnapshot purging log to 0 for term 1 01:21:31,850 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: onUnInitializedFollowerSnapshotReply: UnInitializedFollowerSnapshotReply [followerId=new-server] 01:21:31,850 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]] 01:21:31,850 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Persist data index=2 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:31,850 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=2, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:31,851 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=79f26a2b-dbc1-41cd-b85c-2126309f2ae6}, logIndex: 2, isSendImmediate: true 01:21:31,851 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:31,851 PM [leader-%d23319] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-7448345448504681977/leader/snapshot-0000000068B9925B-32BA7EDE.tmp 01:21:31,851 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: 0 01:21:31,851 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 2 is present for follower new-server 01:21:31,851 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=1, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:31,851 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server131, votingMember=true] 01:21:31,851 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - new-server (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=1, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=2, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:31,851 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - new-server (Follower): Number of entries to be appended = 1 01:21:31,851 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - new-server (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 01:21:31,851 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - new-server (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:31,851 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - new-server: Append log entry and persist JournaledLogEntry{index=2, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:31,852 PM [new-server-%d689516] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-7448345448504681977/new-server/snapshot-0000000068B9925B-32C72681.tmp 01:21:31,852 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - new-server: Updated server config: isVoting: true, peers: [PeerInfo [id=leader, address=null, votingState=VOTING]] 01:21:31,852 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - new-server (Follower): Log size is now 1 01:21:31,852 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - new-server (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=new-server, logLastIndex=2, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=true, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:31,852 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 1 (of expected 1) 01:21:31,852 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for new-server updated: matchIndex: 2, nextIndex: 3 01:21:31,852 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - new-server: awaiting more completions to resolve 1 synchronized callback(s) 01:21:31,852 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 0 01:21:31,855 PM [leader-%d23319] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-7448345448504681977/leader/snapshot-0000000068B9925B-32BA7EDE.v1 01:21:31,855 PM [leader-%d23319] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-7448345448504681977/leader/snapshot-0000000068B9925B-323890A7.v1 01:21:31,856 PM [leader-%d23319] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-7448345448504681977/leader/snapshot-0000000068B9925B-32BA7EDE.v1 01:21:31,856 PM [leader-%d23319] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7448345448504681977/leader/TermInfo.properties 01:21:31,856 PM [leader-%d23319] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7448345448504681977/leader/journal-v1-1.log 01:21:31,856 PM [leader-%d23319] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7448345448504681977/leader/metadata-v1-1.log 01:21:31,856 PM [leader-%d23319] [DEBUG] RaftStorage - leader: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:31.842567847Z, file=/tmp/junit-7448345448504681977/leader/snapshot-0000000068B9925B-323890A7.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:31,856 PM [leader-%d23319] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:31.851082974Z, file=/tmp/junit-7448345448504681977/leader/snapshot-0000000068B9925B-32BA7EDE.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:31,856 PM [leader-%d23319] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:31,856 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:31,856 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=2, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:31,856 PM [new-server-%d689516] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-7448345448504681977/new-server/snapshot-0000000068B9925B-32C72681.v1 01:21:31,856 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 01:21:31,856 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 2 01:21:31,856 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from 1 to 2 01:21:31,856 PM [new-server-%d689516] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-7448345448504681977/new-server/snapshot-0000000068B9925B-3284E9A8.v1 01:21:31,856 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Applying state for log index 2 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:31,856 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - leader: fakeSnapshot purging log to 1 for term 1 01:21:31,857 PM [new-server-%d689516] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-7448345448504681977/new-server/snapshot-0000000068B9925B-32C72681.v1 01:21:31,856 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:31,857 PM [new-server-%d689516] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7448345448504681977/new-server/TermInfo.properties 01:21:31,857 PM [new-server-%d689516] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7448345448504681977/new-server/journal-v1-1.log 01:21:31,857 PM [new-server-%d689516] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7448345448504681977/new-server/metadata-v1-1.log 01:21:31,857 PM [new-server-%d689516] [DEBUG] RaftStorage - new-server: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:31.847571368Z, file=/tmp/junit-7448345448504681977/new-server/snapshot-0000000068B9925B-3284E9A8.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:31,857 PM [new-server-%d689516] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:31.851912321Z, file=/tmp/junit-7448345448504681977/new-server/snapshot-0000000068B9925B-32C72681.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:31,857 PM [new-server-%d689516] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:31,857 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:31,857 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - new-server: persist complete JournaledLogEntry{index=2, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:31,857 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - new-server: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@3256a6e} 01:21:31,857 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - new-server: synchronized callbacks after 4.818 ms [WARN] [09/04/2025 13:21:31.857] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/new-server131] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=true] 01:21:31,953 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): Commit index set to 2 01:21:31,953 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 2 (of expected 1) 01:21:31,953 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 2 01:21:31,953 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - new-server (Follower): Setting last applied to 2 01:21:31,953 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from 1 to 2 01:21:31,953 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - new-server: Applying state for log index 2 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:31,993 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithNoExistingFollower ending 01:21:31,993 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower129] 01:21:32,4 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector130#745524594] 01:21:32,5 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server131] 01:21:32,5 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#283,new-server-writer-45]/waiting 01:21:32,5 PM [new-server-writer-45] [DEBUG] JournalWriteTask - new-server: received 1 commands after 175.6 ms 01:21:32,5 PM [new-server-writer-45] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 33.38 μs 01:21:32,5 PM [new-server-writer-45] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:32,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:32,6 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:32,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:32,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:32,6 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader132] 01:21:32,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#284,leader-writer-46]/waiting 01:21:32,6 PM [leader-writer-46] [DEBUG] JournalWriteTask - leader: received 1 commands after 171.1 ms 01:21:32,6 PM [leader-writer-46] [DEBUG] JournalWriteTask - leader: completed 1 commands in 19.19 μs 01:21:32,7 PM [leader-writer-46] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:32,7 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:32,7 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader: journal closed 01:21:32,7 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:32,7 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:32,7 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leaderCollector133] 01:21:32,12 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeToVotingWithNoLeaderAndForwardedToOtherNodeAfterElectionTimeout starting 01:21:32,15 PM [main] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=1, votedFor=node1 01:21:32,16 PM [main] [DEBUG] EntryJournalV1 - node1: metadata journal open: replayFrom=1 applyTo=0 01:21:32,16 PM [main] [DEBUG] EntryJournalV1 - node1: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:32,19 PM [main] [DEBUG] EntryJournalV1 - node1: journal closed 01:21:32,21 PM [main] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=1, votedFor=node1 01:21:32,21 PM [main] [DEBUG] EntryJournalV1 - node2: metadata journal open: replayFrom=1 applyTo=0 01:21:32,22 PM [main] [DEBUG] EntryJournalV1 - node2: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:32,26 PM [main] [DEBUG] EntryJournalV1 - node2: updated meta to JournalMeta[replayFrom=1, applyTo=2] 01:21:32,26 PM [main] [DEBUG] EntryJournalV1 - node2: journal closed 01:21:32,27 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/node1] [WARN] [09/04/2025 13:21:32.028] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/node1]: org.opendaylight.controller.cluster.common.actor.Monitor@ab29655 01:21:32,29 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - node1: Starting recovery with journal batch size 1000 01:21:32,29 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node1: started executor 01:21:32,29 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node1: started executor 01:21:32,30 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - node1: metadata journal open: replayFrom=1 applyTo=0 01:21:32,31 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - node1: journal open: firstIndex=1 lastIndex=1 replayFrom=1 fileEntries=0 01:21:32,31 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - node1: journal open: applyTo=0 01:21:32,31 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - node1: starting journal recovery 01:21:32,31 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: journal writer started 01:21:32,31 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: waiting to receive commands 01:21:32,31 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=1, votedFor=node1 01:21:32,31 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-5777199355014590110/node1/TermInfo.properties 01:21:32,31 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-5777199355014590110/node1/journal-v1-1.log 01:21:32,31 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-5777199355014590110/node1/metadata-v1-1.log 01:21:32,31 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node1: no eligible files found 01:21:32,31 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from -1 to -1 01:21:32,31 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - node1: applying entries up to 0 01:21:32,32 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - node1: recovered journal DefaultLogEntry[index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=false]]}] 01:21:32,32 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - node1: Updated server config: isVoting: false, peers: [PeerInfo [id=node2, address=null, votingState=NON_VOTING]] 01:21:32,32 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from -1 to -1 01:21:32,32 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - node1: Recovery completed in in 1.493 ms: last log index = 0, last log term = 1, napshot index = -1, snapshot term = -1, journal size = 1 01:21:32,32 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - node1: Recovery completed and cannot restore from snapshot 01:21:32,34 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/node2] [WARN] [09/04/2025 13:21:32.034] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/node2]: org.opendaylight.controller.cluster.common.actor.Monitor@4e6f9c22 01:21:32,34 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - node2: Starting recovery with journal batch size 1000 01:21:32,35 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node2: started executor 01:21:32,35 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node2: started executor 01:21:32,35 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - node2: metadata journal open: replayFrom=1 applyTo=2 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - node2: journal open: firstIndex=1 lastIndex=2 replayFrom=1 fileEntries=0 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - node2: journal open: applyTo=2 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - node2: starting journal recovery 01:21:32,36 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: journal writer started 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=1, votedFor=node1 01:21:32,36 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: waiting to receive commands 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-5777199355014590110/node2/TermInfo.properties 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-5777199355014590110/node2/journal-v1-1.log 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-5777199355014590110/node2/metadata-v1-1.log 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node2: no eligible files found 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from -1 to -1 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - node2: applying entries up to 2 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - node2: recovered journal DefaultLogEntry[index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=false]]}] 01:21:32,36 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - node2: Updated server config: isVoting: false, peers: [PeerInfo [id=node1, address=null, votingState=NON_VOTING]] 01:21:32,37 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] JournalRecovery - node2: recovered journal DefaultLogEntry[index=1, term=1, command=2] 01:21:32,37 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from -1 to 1 01:21:32,37 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - node2: Recovery completed in in 1.146 ms: last log index = 1, last log term = 1, napshot index = -1, snapshot term = -1, journal size = 2 01:21:32,37 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - node2: Recovery completed and cannot restore from snapshot 01:21:32,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - node1: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={node1=true, node2=true}, serversVisited=[]], state: Idle 01:21:32,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:32,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - node1: Sending local ElectionTimeout to start leader election 01:21:32,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - node1: Updated server config: isVoting: true, peers: [PeerInfo [id=node2, address=null, votingState=VOTING]] 01:21:32,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:32,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node1 (Follower): Received TimeoutNow - switching to Candidate 01:21:32,48 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - node1 (Candidate): Election: Candidate has following voting peers: [node2] 01:21:32,51 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=2, votedFor=node1 01:21:32,51 PM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - node1 (Candidate): Starting new election term 2 01:21:32,51 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - node1 (Candidate): Sending RequestVote{term=2, candidateId=node1, lastLogIndex=0, lastLogTerm=1} to peer node2 01:21:32,51 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - node1 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:21:32,51 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node2 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:21:32,51 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node2 (Follower): Cluster state: [] 01:21:32,51 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node2 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=node1, lastLogIndex=0, lastLogTerm=1} with higher term appears reachable, updating term. 01:21:32,51 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - node2 (Follower): Term 2 in "RequestVote{term=2, candidateId=node1, lastLogIndex=0, lastLogTerm=1}" message is greater than follower's term 1 - updating term 01:21:32,53 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=2, votedFor=null 01:21:32,53 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node2 (Follower): In requestVote: RequestVote{term=2, candidateId=node1, lastLogIndex=0, lastLogTerm=1} - currentTerm: 2, votedFor: null, lastIndex: 1, lastTerm: 1 01:21:32,53 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node2 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=false} 01:21:32,53 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - node1 (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=false}, my term is 2 01:21:32,53 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - node1 (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=false}, current voteCount: 1 01:21:32,363 PM [test-pekko.actor.default-dispatcher-12] [WARN] RaftActorVotingConfigSupport - node1: Leader election timed out - cannot apply operation ChangeServersVotingStatus [serverVotingStatusMap={node1=true, node2=true}, serversVisited=[]] 01:21:32,363 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - node1: Updated server config: isVoting: false, peers: [PeerInfo [id=node2, address=pekko://test/user/node2, votingState=NON_VOTING]] 01:21:32,363 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - node1: tryToForwardOperationToAnotherServer - servers already visited [] 01:21:32,363 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - node1: Found server ActorSelection[Anchor(pekko://test/), Path(/user/node2)] to forward to 01:21:32,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - node2: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={node1=true, node2=true}, serversVisited=[node1]], state: Idle 01:21:32,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:32,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - node2: Sending local ElectionTimeout to start leader election 01:21:32,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - node2: Updated server config: isVoting: true, peers: [PeerInfo [id=node1, address=null, votingState=VOTING]] 01:21:32,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node2 (Follower): Received TimeoutNow - switching to Candidate 01:21:32,364 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - node2 (Candidate): Election: Candidate has following voting peers: [node1] 01:21:32,367 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=3, votedFor=node2 01:21:32,367 PM [test-pekko.actor.default-dispatcher-12] [INFO] Candidate - node2 (Candidate): Starting new election term 3 01:21:32,367 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Candidate - node2 (Candidate): Sending RequestVote{term=3, candidateId=node2, lastLogIndex=1, lastLogTerm=1} to peer node1 01:21:32,367 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorBehavior - node2 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:21:32,367 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - node1 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:21:32,367 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - node1 (Follower): Cluster state: [] 01:21:32,367 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - node1 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=node2, lastLogIndex=1, lastLogTerm=1} with higher term appears reachable, updating term. 01:21:32,367 PM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - node1 (Follower): Term 3 in "RequestVote{term=3, candidateId=node2, lastLogIndex=1, lastLogTerm=1}" message is greater than follower's term 2 - updating term 01:21:32,370 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=3, votedFor=null 01:21:32,370 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - node1 (Follower): In requestVote: RequestVote{term=3, candidateId=node2, lastLogIndex=1, lastLogTerm=1} - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 1 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=3, votedFor=node2 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - node1 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - node2 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - node2 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Election: Leader has following peers: [node1] 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Checking sendAppendEntries for follower node1: active: false, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - node2 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 01:21:32,373 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - node1: Last sync leader does not match current leader node2, need to catch up to -1 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - node2: New leader node2 elected 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - node2: New server configuration : [ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]] 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - node2: Persist data index=2 term=3 command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]} 01:21:32,373 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - node2: Append log entry and persist JournaledLogEntry{index=2, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=true} [WARN] [09/04/2025 13:21:32.373] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/node1] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=node1, initialSyncDone=false] 01:21:32,374 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=49cf4703-38ac-442f-bd25-24a95655aca4}, logIndex: 2, isSendImmediate: true 01:21:32,374 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Checking sendAppendEntries for follower node1: active: false, followerNextIndex: 1, leaderLastIndex: 2, leaderSnapShotIndex: -1 01:21:32,374 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - node2: Returning OK for operation ChangeServersVotingStatus [serverVotingStatusMap={node1=true, node2=true}, serversVisited=[node1]] 01:21:32,374 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: received 1 commands after 337.9 ms 01:21:32,374 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): handleAppendEntriesReply - FollowerLogInformation for node1 updated: matchIndex: 0, nextIndex: 1 01:21:32,374 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - node2: fakeSnapshot purging log to 0 for term 3 01:21:32,374 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Checking sendAppendEntries for follower node1: active: true, followerNextIndex: 1, leaderLastIndex: 2, leaderSnapShotIndex: 0 01:21:32,374 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): sendAppendEntries: 1 is present for follower node1 01:21:32,374 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - node2 (Leader): Sending AppendEntries to follower node1: AppendEntries{term=3, leaderId=node2, prevLogIndex=0, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=1, command=2, pending=false}, JournaledLogEntry{index=2, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=true}]} 01:21:32,374 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - node1: Lagging 0 entries behind leader node2 and reached -1 (of expected -1) 01:21:32,375 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - node2 (Leader): sendAppendEntries: 1 is present for follower node1 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=node2, prevLogIndex=0, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=1, command=2, pending=false}, JournaledLogEntry{index=2, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=true}]} 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): Number of entries to be appended = 2 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): Append entry to log 2 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - node1: Append log entry and persist JournaledLogEntry{index=1, term=1, command=2, pending=false} 01:21:32,375 PM [node2-writer-48] [DEBUG] RaftStorageCompleter - node2: 1 completion(s) pending 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]} 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - node1: Append log entry and persist JournaledLogEntry{index=2, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=true} 01:21:32,375 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - node2: running 1 completion(s) 01:21:32,375 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - node2: persist complete JournaledLogEntry{index=2, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=true} 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - node1: Updated server config: isVoting: true, peers: [PeerInfo [id=node2, address=pekko://test/user/node2, votingState=VOTING]] 01:21:32,375 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: completed 1 commands in 1.189 ms 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): Log size is now 3 01:21:32,375 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: waiting to receive commands 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): Commit index set to 1 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=node1, logLastIndex=2, logLastTerm=3, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - node1: Lagging 0 entries behind leader node2 and reached 1 (of expected -1) 01:21:32,375 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: received 2 commands after 344.4 ms 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): -1, lastIndex(): 2 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node1 (Follower): Setting last applied to 0 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from -1 to 0 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - node1: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=false]]} 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node1 (Follower): Setting last applied to 1 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from 0 to 1 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - node1: Applying state for log index 1 data 2 01:21:32,375 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - node1: applyState called: 2 01:21:32,375 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - node2 (Leader): handleAppendEntriesReply - FollowerLogInformation for node1 updated: matchIndex: 2, nextIndex: 3 01:21:32,376 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - node2 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 01:21:32,376 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - node2 (Leader): Setting last applied to 2 01:21:32,376 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from 1 to 2 01:21:32,376 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - node2: Applying state for log index 2 data VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]} 01:21:32,376 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - node1: fakeSnapshot purging log to 0 for term 3 01:21:32,376 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - node2: fakeSnapshot purging log to 1 for term 3 01:21:32,376 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - node1: awaiting more completions to resolve 2 synchronized callback(s) 01:21:32,376 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - node2 (Leader): Checking sendAppendEntries for follower node1: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 1 01:21:32,376 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: received 1 commands after 585.5 μs 01:21:32,376 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorVotingConfigSupport - node2: VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:32,376 PM [node2-writer-48] [DEBUG] EntryJournalV1 - node2: updated meta to JournalMeta[replayFrom=1, applyTo=3] 01:21:32,376 PM [node2-writer-48] [DEBUG] RaftStorageCompleter - node2: 1 completion(s) pending 01:21:32,376 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: completed 1 commands in 406.5 μs 01:21:32,376 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - node2: running 1 completion(s) 01:21:32,376 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: waiting to receive commands 01:21:32,377 PM [node1-writer-47] [DEBUG] RaftStorageCompleter - node1: 2 completion(s) pending 01:21:32,377 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: completed 2 commands in 2.044 ms 01:21:32,377 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: received 1 commands after 5.600 μs 01:21:32,377 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - node1: running 2 completion(s) 01:21:32,377 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - node1: persist complete JournaledLogEntry{index=1, term=1, command=2, pending=false} 01:21:32,377 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - node1: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=1, command=2, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@39809780} 01:21:32,377 PM [node1-writer-47] [DEBUG] EntryJournalV1 - node1: updated meta to JournalMeta[replayFrom=1, applyTo=2] 01:21:32,377 PM [node1-writer-47] [DEBUG] RaftStorageCompleter - node1: 1 completion(s) pending 01:21:32,377 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: completed 1 commands in 160.2 μs 01:21:32,377 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - node1: persist complete JournaledLogEntry{index=2, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=false} 01:21:32,378 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: waiting to receive commands 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - node1: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=2, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@39809780} 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - node1: synchronized callbacks after 1.967 ms 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - node1: running 1 completion(s) 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): Commit index set to 2 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - node1: Lagging 0 entries behind leader node2 and reached 2 (of expected -1) 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - node1 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 1, lastIndex(): 2 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node1 (Follower): Setting last applied to 2 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from 1 to 2 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - node1: Applying state for log index 2 data VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=true], ServerInfo[peerId=node2, isVoting=true]]} 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - node1: fakeSnapshot purging log to 1 for term 3 01:21:32,378 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: received 1 commands after 600.6 μs 01:21:32,378 PM [node1-writer-47] [DEBUG] EntryJournalV1 - node1: updated meta to JournalMeta[replayFrom=1, applyTo=3] 01:21:32,378 PM [node1-writer-47] [DEBUG] RaftStorageCompleter - node1: 1 completion(s) pending 01:21:32,378 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - node1: running 1 completion(s) 01:21:32,378 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: completed 1 commands in 139.7 μs 01:21:32,378 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: waiting to receive commands [WARN] [09/04/2025 13:21:32.378] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/node1] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=node1, initialSyncDone=true] 01:21:32,493 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - node1: Lagging 0 entries behind leader node2 and reached 2 (of expected -1) 01:21:32,576 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeToVotingWithNoLeaderAndForwardedToOtherNodeAfterElectionTimeout ending 01:21:32,577 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower134] 01:21:32,588 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector135#893712370] 01:21:32,589 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/node1] 01:21:32,589 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - node1: terminating thread VirtualThread[#290,node1-writer-47]/waiting 01:21:32,589 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: received 1 commands after 210.6 ms 01:21:32,589 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: completed 1 commands in 37.46 μs 01:21:32,589 PM [node1-writer-47] [DEBUG] JournalWriteTask - node1: journal writer stopped 01:21:32,590 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - node1: journal closed 01:21:32,590 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - node1: journal closed 01:21:32,590 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node1: stopped executor with 0 remaining tasks 01:21:32,590 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node1: stopped executor with 0 remaining tasks 01:21:32,590 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector136#-1908839434] 01:21:32,590 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/node2] 01:21:32,591 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - node2: terminating thread VirtualThread[#291,node2-writer-48]/waiting 01:21:32,591 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: received 1 commands after 214.6 ms 01:21:32,591 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: completed 1 commands in 20.51 μs 01:21:32,591 PM [node2-writer-48] [DEBUG] JournalWriteTask - node2: journal writer stopped 01:21:32,591 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - node2: journal closed 01:21:32,591 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - node2: journal closed 01:21:32,591 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node2: stopped executor with 0 remaining tasks 01:21:32,591 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - node2: stopped executor with 0 remaining tasks 01:21:32,697 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithNoConsensusReached starting 01:21:32,699 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server139] [WARN] [09/04/2025 13:21:32.699] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server139]: org.opendaylight.controller.cluster.common.actor.Monitor@728d564 01:21:32,700 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:32,700 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: started executor 01:21:32,700 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: started executor 01:21:32,701 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:32,702 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:32,702 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:32,702 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:32,702 PM [new-server-writer-49] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:32,702 PM [new-server-writer-49] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:32,702 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/junit-509615374858259102/new-server/TermInfo.properties does not exist 01:21:32,705 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:32,705 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:32,705 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-509615374858259102/new-server/TermInfo.properties 01:21:32,705 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-509615374858259102/new-server/journal-v1-1.log 01:21:32,705 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-509615374858259102/new-server/metadata-v1-1.log 01:21:32,706 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:32,706 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:32,706 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - new-server: Recovery completed in in 3.702 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:32,706 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:32,706 PM [test-pekko.actor.default-dispatcher-12] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:32,707 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader140] [WARN] [09/04/2025 13:21:32.707] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader140]: org.opendaylight.controller.cluster.common.actor.Monitor@422a5be5 01:21:32,708 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:32,708 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:32,708 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: started executor 01:21:32,708 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: started executor 01:21:32,708 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:32,709 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:32,709 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:32,709 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:32,709 PM [leader-writer-50] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:32,709 PM [leader-writer-50] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:32,709 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/junit-509615374858259102/leader/TermInfo.properties does not exist 01:21:32,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:32,713 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:32,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-509615374858259102/leader/TermInfo.properties 01:21:32,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-509615374858259102/leader/journal-v1-1.log 01:21:32,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-509615374858259102/leader/metadata-v1-1.log 01:21:32,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: no eligible files found 01:21:32,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:32,713 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - leader: Recovery completed in in 3.781 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:32,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:32,713 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [] 01:21:32,724 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server139, votingMember=true], state: Idle 01:21:32,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server139, votingMember=true] 01:21:32,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 01:21:32,725 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on new-server 01:21:32,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader: last applied journal index prior to capture: 0 01:21:32,725 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader: support() called 01:21:32,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating capture snapshot for new server new-server 01:21:32,725 PM [leader-%d685737] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:32,726 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:32,726 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader: Persising snapshot at EntryInfo[index=-1, term=-1]/EntryInfo[index=-1, term=-1] 01:21:32,726 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader: support() called 01:21:32,726 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 01:21:32,726 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): sendInstallSnapshot 01:21:32,726 PM [leader-%d685738] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-509615374858259102/leader/snapshot-0000000068B9925C-2B4BDA78.tmp 01:21:32,726 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Snapshot 157 bytes, total chunks to send: 1 01:21:32,726 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Next chunk: total length=157, offset=0, size=157, hashCode=1546036366 01:21:32,726 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): next snapshot chunk size for follower new-server: 157 01:21:32,726 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): InstallSnapshot sent to follower Vector(user, new-server139), Chunk: 1/1 01:21:32,726 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - new-server (Follower): Term 1 in "InstallSnapshot{term=1, leaderId=leader, lastIncludedIndex=-1, lastIncludedTerm=-1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1]}" message is greater than follower's term 0 - updating term 01:21:32,729 PM [leader-%d685738] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-509615374858259102/leader/snapshot-0000000068B9925C-2B4BDA78.v1 01:21:32,729 PM [leader-%d685738] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-509615374858259102/leader/snapshot-0000000068B9925C-2B4BDA78.v1 01:21:32,729 PM [leader-%d685738] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-509615374858259102/leader/TermInfo.properties 01:21:32,729 PM [leader-%d685738] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-509615374858259102/leader/journal-v1-1.log 01:21:32,729 PM [leader-%d685738] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-509615374858259102/leader/metadata-v1-1.log 01:21:32,729 PM [leader-%d685738] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:32.726391416Z, file=/tmp/junit-509615374858259102/leader/snapshot-0000000068B9925C-2B4BDA78.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:32,729 PM [leader-%d685738] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:32,729 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:32,730 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: snapshot is durable as of 2025-09-04T13:21:32.726391416Z 01:21:32,730 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=1, votedFor=null 01:21:32,730 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: Snapshot success, discarding journal entries up to 0 01:21:32,730 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleInstallSnapshot: InstallSnapshot{term=1, leaderId=leader, lastIncludedIndex=-1, lastIncludedTerm=-1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1]} 01:21:32,730 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorSnapshotMessageSupport - leader: SnapshotComplete received 01:21:32,730 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - new-server: Last sync leader does not match current leader leader, need to catch up to -1 01:21:32,730 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotTracker - new-server (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 01:21:32,730 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - new-server (Follower): Snapshot received from leader: leader 01:21:32,730 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - new-server: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=157}} 01:21:32,730 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - new-server: support() called [WARN] [09/04/2025 13:21:32.730] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/new-server139] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=false] 01:21:32,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - new-server: Converted InstallSnapshot from leader: leader to state 01:21:32,731 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - new-server: last applied journal index prior to persisting applied snapshot: 0 01:21:32,731 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - new-server: support() called 01:21:32,731 PM [new-server-%d705241] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-509615374858259102/new-server/snapshot-0000000068B9925C-2B9AF562.tmp 01:21:32,733 PM [new-server-%d705241] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-509615374858259102/new-server/snapshot-0000000068B9925C-2B9AF562.v1 01:21:32,733 PM [new-server-%d705241] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-509615374858259102/new-server/snapshot-0000000068B9925C-2B9AF562.v1 01:21:32,733 PM [new-server-%d705241] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-509615374858259102/new-server/TermInfo.properties 01:21:32,733 PM [new-server-%d705241] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-509615374858259102/new-server/journal-v1-1.log 01:21:32,733 PM [new-server-%d705241] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-509615374858259102/new-server/metadata-v1-1.log 01:21:32,733 PM [new-server-%d705241] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:32.731575650Z, file=/tmp/junit-509615374858259102/new-server/snapshot-0000000068B9925C-2B9AF562.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:32,733 PM [new-server-%d705241] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:32,733 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:32,733 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - new-server: snapshot is durable as of 2025-09-04T13:21:32.731575650Z 01:21:32,734 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - new-server: support() called 01:21:32,734 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - new-server: applySnapshot called 01:21:32,734 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=1, followerId=new-server, chunkIndex=1, success=true} 01:21:32,734 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - new-server: Snapshot success, discarding journal entries up to 0 01:21:32,734 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorSnapshotMessageSupport - new-server: SnapshotComplete received 01:21:32,734 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=1, followerId=new-server, chunkIndex=1, success=true} 01:21:32,734 PM [test-pekko.actor.default-dispatcher-13] [INFO] AbstractLeader - leader (Leader): Snapshot successfully installed on follower new-server (last chunk 1) - matchIndex set to -1, nextIndex set to 0 01:21:32,734 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - Sent message UnInitializedFollowerSnapshotReply to self 01:21:32,775 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onUnInitializedFollowerSnapshotReply: UnInitializedFollowerSnapshotReply [followerId=new-server] 01:21:32,775 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]] 01:21:32,775 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Persist data index=0 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:32,775 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:32,775 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=c3a5dba9-0ccd-4a94-a24d-0c82e42b9f67}, logIndex: 0, isSendImmediate: true 01:21:32,775 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:32,775 PM [leader-%d685739] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-509615374858259102/leader/snapshot-0000000068B9925C-2E39FDBA.tmp 01:21:32,775 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:32,775 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:32,775 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:32,776 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server139, votingMember=true] 01:21:32,777 PM [leader-%d685739] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-509615374858259102/leader/snapshot-0000000068B9925C-2E39FDBA.v1 01:21:32,778 PM [leader-%d685739] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-509615374858259102/leader/snapshot-0000000068B9925C-2B4BDA78.v1 01:21:32,778 PM [leader-%d685739] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-509615374858259102/leader/snapshot-0000000068B9925C-2E39FDBA.v1 01:21:32,778 PM [leader-%d685739] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-509615374858259102/leader/TermInfo.properties 01:21:32,778 PM [leader-%d685739] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-509615374858259102/leader/journal-v1-1.log 01:21:32,778 PM [leader-%d685739] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-509615374858259102/leader/metadata-v1-1.log 01:21:32,778 PM [leader-%d685739] [DEBUG] RaftStorage - leader: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:32.726391416Z, file=/tmp/junit-509615374858259102/leader/snapshot-0000000068B9925C-2B4BDA78.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:32,778 PM [leader-%d685739] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:32.775552442Z, file=/tmp/junit-509615374858259102/leader/snapshot-0000000068B9925C-2E39FDBA.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:32,778 PM [leader-%d685739] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:32,778 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:32,778 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:32,825 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server2, newServerAddress=, votingMember=false], state: Persisting 01:21:32,825 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Server operation already in progress - queueing AddServer [newServerId=new-server2, newServerAddress=, votingMember=false] 01:21:32,842 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:32,963 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:32,963 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:33,83 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:33,83 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:33,203 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:33,203 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:33,322 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:33,322 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:33,443 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:33,443 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:33,562 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:33,562 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:33,683 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:33,683 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:34,792 PM [test-pekko.actor.default-dispatcher-12] [WARN] RaftActorVotingConfigSupport - leader: Timeout occured while replicating the new server configuration for new-server 01:21:34,792 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Returning PRIOR_REQUEST_CONSENSUS_TIMEOUT for operation AddServer [newServerId=new-server2, newServerAddress=, votingMember=false] 01:21:34,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server2, newServerAddress=, votingMember=false], state: Persisting 01:21:34,793 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Returning PRIOR_REQUEST_CONSENSUS_TIMEOUT for operation AddServer [newServerId=new-server2, newServerAddress=, votingMember=false] 01:21:34,793 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithNoConsensusReached ending 01:21:34,794 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower137] 01:21:34,805 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector138#1488336819] 01:21:34,806 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server139] 01:21:34,806 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#294,new-server-writer-49]/waiting 01:21:34,806 PM [new-server-writer-49] [DEBUG] JournalWriteTask - new-server: received 1 commands after 2.104 s 01:21:34,806 PM [new-server-writer-49] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 41.15 μs 01:21:34,806 PM [new-server-writer-49] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:34,807 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:34,807 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:34,807 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:34,807 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:34,807 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader140] 01:21:34,808 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#295,leader-writer-50]/waiting 01:21:34,808 PM [leader-writer-50] [DEBUG] JournalWriteTask - leader: received 1 commands after 2.099 s 01:21:34,808 PM [leader-writer-50] [DEBUG] JournalWriteTask - leader: completed 1 commands in 21.68 μs 01:21:34,808 PM [leader-writer-50] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:34,808 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:34,808 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader: journal closed 01:21:34,808 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:34,808 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:34,809 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leaderCollector141] 01:21:34,809 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-serverCollector142] 01:21:34,814 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServerLeaderWithNoFollowers starting 01:21:34,815 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader144] 01:21:34,816 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:34,816 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:34,816 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: started executor 01:21:34,816 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: started executor [WARN] [09/04/2025 13:21:34.816] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader144]: org.opendaylight.controller.cluster.common.actor.Monitor@1a14cf7a 01:21:34,820 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:34,821 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:34,821 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:34,821 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:34,821 PM [leader-writer-51] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:34,821 PM [leader-writer-51] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:34,821 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/junit-7542236933626684606/leader/TermInfo.properties does not exist 01:21:34,825 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:34,825 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:34,825 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7542236933626684606/leader/TermInfo.properties 01:21:34,825 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7542236933626684606/leader/journal-v1-1.log 01:21:34,825 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7542236933626684606/leader/metadata-v1-1.log 01:21:34,825 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: no eligible files found 01:21:34,825 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:34,825 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - leader: Recovery completed in in 4.436 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:34,825 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:34,826 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [] 01:21:34,837 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: onRemoveServer: RemoveServer{serverId='leader'}, state: Idle 01:21:34,838 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServerLeaderWithNoFollowers ending 01:21:34,838 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower143] 01:21:34,850 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader144] 01:21:34,850 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#300,leader-writer-51]/waiting 01:21:34,851 PM [leader-writer-51] [DEBUG] JournalWriteTask - leader: received 1 commands after 29.60 ms 01:21:34,851 PM [leader-writer-51] [DEBUG] JournalWriteTask - leader: completed 1 commands in 38.83 μs 01:21:34,851 PM [leader-writer-51] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:34,852 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:34,852 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader: journal closed 01:21:34,852 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:34,852 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:34,857 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithLeaderChangeBeforePriorSnapshotComplete starting 01:21:34,858 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server147] [WARN] [09/04/2025 13:21:34.858] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server147]: org.opendaylight.controller.cluster.common.actor.Monitor@29e4bb6c 01:21:34,860 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:34,860 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: started executor 01:21:34,860 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: started executor 01:21:34,861 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:34,861 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:34,862 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:34,862 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:34,862 PM [new-server-writer-52] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:34,862 PM [new-server-writer-52] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:34,862 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/junit-7616272768599826562/new-server/TermInfo.properties does not exist 01:21:34,865 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:34,865 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:34,865 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7616272768599826562/new-server/TermInfo.properties 01:21:34,865 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7616272768599826562/new-server/journal-v1-1.log 01:21:34,865 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-7616272768599826562/new-server/metadata-v1-1.log 01:21:34,865 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:34,865 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:34,865 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - new-server: Recovery completed in in 3.411 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:34,865 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:34,866 PM [test-pekko.actor.default-dispatcher-12] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:34,866 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader148] [WARN] [09/04/2025 13:21:34.867] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader148]: org.opendaylight.controller.cluster.common.actor.Monitor@4abec139 01:21:34,868 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:34,868 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:34,868 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:34,868 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:34,869 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:34,869 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:34,869 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:34,869 PM [leader-writer-53] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:34,870 PM [leader-writer-53] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:34,870 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:34,870 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-7616272768599826562/leader/TermInfo.properties does not exist 01:21:34,872 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:34,872 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:34,872 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7616272768599826562/leader/TermInfo.properties 01:21:34,872 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7616272768599826562/leader/journal-v1-1.log 01:21:34,872 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7616272768599826562/leader/metadata-v1-1.log 01:21:34,872 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: no eligible files found 01:21:34,872 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:34,873 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Recovery completed in in 2.865 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:34,873 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:34,873 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [] 01:21:34,875 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Take a snapshot of current state. lastReplicatedLog is null and replicatedToAllIndex is -1 01:21:34,875 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 01:21:34,875 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=true] 01:21:34,875 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: captured snapshot at lastSequenceNumber: 0 01:21:34,875 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Persising snapshot at EntryInfo[index=-1, term=-1]/EntryInfo[index=-1, term=-1] 01:21:34,875 PM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - leader: support() called 01:21:34,876 PM [leader-%d267043] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-7616272768599826562/leader/snapshot-0000000068B9925E-3439A96C.tmp 01:21:34,876 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: user triggered or root overwrite snapshot encountered, trimming log up to last applied index -1 01:21:34,876 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 01:21:34,880 PM [leader-%d267043] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-7616272768599826562/leader/snapshot-0000000068B9925E-3439A96C.v1 01:21:34,880 PM [leader-%d267043] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-7616272768599826562/leader/snapshot-0000000068B9925E-3439A96C.v1 01:21:34,880 PM [leader-%d267043] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7616272768599826562/leader/TermInfo.properties 01:21:34,880 PM [leader-%d267043] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7616272768599826562/leader/journal-v1-1.log 01:21:34,880 PM [leader-%d267043] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-7616272768599826562/leader/metadata-v1-1.log 01:21:34,880 PM [leader-%d267043] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:34.876194156Z, file=/tmp/junit-7616272768599826562/leader/snapshot-0000000068B9925E-3439A96C.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:34,880 PM [leader-%d267043] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:34,880 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:34,886 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server147, votingMember=true], state: Idle 01:21:34,886 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server147, votingMember=true] 01:21:34,886 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: captureToInstall should not be called in state PersistCapture[lastJournalIndex=0, request=CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=true]] 01:21:34,886 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Snapshot already in progress - waiting for completion 01:21:34,989 PM [main] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:34,989 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:34,989 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: snapshot is durable as of 2025-09-04T13:21:34.876194156Z 01:21:34,989 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: Snapshot success, discarding journal entries up to 0 01:21:34,989 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onSnapshotComplete 01:21:34,989 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: No longer the leader 01:21:34,990 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorSnapshotMessageSupport - leader: SnapshotComplete received 01:21:34,990 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: handleInstallSnapshotTimeout for new server new-server 01:21:34,990 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Returning NO_LEADER for operation AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server147, votingMember=true] 01:21:34,990 PM [test-pekko.actor.default-dispatcher-13] [WARN] RaftActorVotingConfigSupport - leader: Timeout occured for new server new-server while waiting for prior snapshot to complete 01:21:34,990 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithLeaderChangeBeforePriorSnapshotComplete ending 01:21:34,990 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower145] 01:21:35,2 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector146#1802475107] 01:21:35,2 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server147] 01:21:35,2 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#301,new-server-writer-52]/waiting 01:21:35,2 PM [new-server-writer-52] [DEBUG] JournalWriteTask - new-server: received 1 commands after 140.7 ms 01:21:35,3 PM [new-server-writer-52] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 28.20 μs 01:21:35,3 PM [new-server-writer-52] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:35,3 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:35,3 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:35,3 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:35,3 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:35,3 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader148] 01:21:35,3 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#302,leader-writer-53]/waiting 01:21:35,4 PM [leader-writer-53] [DEBUG] JournalWriteTask - leader: received 1 commands after 134.1 ms 01:21:35,4 PM [leader-writer-53] [DEBUG] JournalWriteTask - leader: completed 1 commands in 19.91 μs 01:21:35,4 PM [leader-writer-53] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:35,4 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:35,4 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal closed 01:21:35,4 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:35,4 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:35,8 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServersAsNonVoting starting 01:21:35,10 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server151] [WARN] [09/04/2025 13:21:35.010] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server151]: org.opendaylight.controller.cluster.common.actor.Monitor@45ef5f8c 01:21:35,11 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:35,11 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: started executor 01:21:35,11 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: started executor 01:21:35,19 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:35,21 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:35,21 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:35,21 PM [new-server-writer-54] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:35,21 PM [new-server-writer-54] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:35,22 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:35,22 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-16008788317331156117/new-server/TermInfo.properties does not exist 01:21:35,25 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:35,25 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:35,25 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16008788317331156117/new-server/TermInfo.properties 01:21:35,25 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16008788317331156117/new-server/journal-v1-1.log 01:21:35,25 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16008788317331156117/new-server/metadata-v1-1.log 01:21:35,25 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:35,25 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:35,26 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - new-server: Recovery completed in in 3.722 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:35,26 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:35,27 PM [test-pekko.actor.default-dispatcher-13] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:35,30 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader152] [WARN] [09/04/2025 13:21:35.031] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader152]: org.opendaylight.controller.cluster.common.actor.Monitor@78f4fe20 01:21:35,33 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:35,33 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:35,33 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:35,33 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:35,33 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:35,34 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:35,34 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:35,34 PM [leader-writer-55] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:35,34 PM [leader-writer-55] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:35,34 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:35,34 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-16008788317331156117/leader/TermInfo.properties does not exist 01:21:35,36 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:35,36 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:35,36 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16008788317331156117/leader/TermInfo.properties 01:21:35,36 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16008788317331156117/leader/journal-v1-1.log 01:21:35,36 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16008788317331156117/leader/metadata-v1-1.log 01:21:35,36 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: no eligible files found 01:21:35,36 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:35,36 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Recovery completed in in 2.399 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:35,36 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:35,37 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [] 01:21:35,61 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server151, votingMember=false], state: Idle 01:21:35,62 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server151, votingMember=false] 01:21:35,62 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: New follower is non-voting - directly persisting new server configuration 01:21:35,62 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]] 01:21:35,62 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Persist data index=0 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,63 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:35,63 PM [leader-%d613342] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-16008788317331156117/leader/snapshot-0000000068B9925F-03C792B9.tmp 01:21:35,64 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=ab03207d-8c05-4368-a5bd-cf0e2c4f7aca}, logIndex: 0, isSendImmediate: true 01:21:35,64 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 0 01:21:35,64 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to 0 01:21:35,64 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,65 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,66 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server151, votingMember=false] 01:21:35,66 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - new-server (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:35,66 PM [leader-%d613342] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-16008788317331156117/leader/snapshot-0000000068B9925F-03C792B9.v1 01:21:35,66 PM [leader-%d613342] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-16008788317331156117/leader/snapshot-0000000068B9925F-03C792B9.v1 01:21:35,67 PM [leader-%d613342] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16008788317331156117/leader/TermInfo.properties 01:21:35,67 PM [leader-%d613342] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16008788317331156117/leader/journal-v1-1.log 01:21:35,67 PM [leader-%d613342] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16008788317331156117/leader/metadata-v1-1.log 01:21:35,67 PM [leader-%d613342] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.063410873Z, file=/tmp/junit-16008788317331156117/leader/snapshot-0000000068B9925F-03C792B9.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:35,67 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:35,67 PM [leader-%d613342] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:35,67 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:35,67 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:35,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=1, votedFor=null 01:21:35,69 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - new-server: Last sync leader does not match current leader leader, need to catch up to -1 01:21:35,69 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for new-server updated: matchIndex: -1, nextIndex: 0 01:21:35,69 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,69 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server [WARN] [09/04/2025 13:21:35.069] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/new-server151] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=false] 01:21:35,70 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): Number of entries to be appended = 1 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - new-server: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - new-server: Updated server config: isVoting: false, peers: [PeerInfo [id=leader, address=null, votingState=VOTING]] 01:21:35,70 PM [new-server-%d990404] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-16008788317331156117/new-server/snapshot-0000000068B9925F-04338DF2.tmp 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): Log size is now 1 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): Commit index set to 0 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=new-server, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=true, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - new-server (Follower): Setting last applied to 0 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to 0 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - new-server: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,70 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - new-server: awaiting more completions to resolve 1 synchronized callback(s) 01:21:35,70 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for new-server updated: matchIndex: 0, nextIndex: 1 01:21:35,71 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,72 PM [new-server-%d990404] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-16008788317331156117/new-server/snapshot-0000000068B9925F-04338DF2.v1 01:21:35,72 PM [new-server-%d990404] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-16008788317331156117/new-server/snapshot-0000000068B9925F-04338DF2.v1 01:21:35,72 PM [new-server-%d990404] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16008788317331156117/new-server/TermInfo.properties 01:21:35,72 PM [new-server-%d990404] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16008788317331156117/new-server/journal-v1-1.log 01:21:35,72 PM [new-server-%d990404] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16008788317331156117/new-server/metadata-v1-1.log 01:21:35,72 PM [new-server-%d990404] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.070487538Z, file=/tmp/junit-16008788317331156117/new-server/snapshot-0000000068B9925F-04338DF2.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:35,72 PM [new-server-%d990404] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:35,72 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:35,72 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - new-server: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:35,72 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - new-server: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@485e717d} 01:21:35,72 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - new-server: synchronized callbacks after 1.939 ms [WARN] [09/04/2025 13:21:35.073] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/new-server151] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=true] 01:21:35,173 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:35,293 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:35,413 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:35,533 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:35,624 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server2, newServerAddress=pekko://test/user/follower149, votingMember=false], state: Idle 01:21:35,624 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server2, newServerAddress=pekko://test/user/follower149, votingMember=false] 01:21:35,624 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: New follower is non-voting - directly persisting new server configuration 01:21:35,624 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]] 01:21:35,624 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Persist data index=1 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,624 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:35,625 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=94a2925e-48dc-4dfe-b061-09dd56a8bdde}, logIndex: 1, isSendImmediate: true 01:21:35,625 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 1 01:21:35,625 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from 0 to 1 01:21:35,625 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - leader: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,625 PM [leader-%d613343] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-16008788317331156117/leader/snapshot-0000000068B9925F-25423887.tmp 01:21:35,625 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:35,625 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 1 is present for follower new-server 01:21:35,625 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:35,625 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server2: active: false, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:35,626 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - new-server2: Last sync leader does not match current leader leader, need to catch up to -1 01:21:35,626 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation AddServer [newServerId=new-server2, newServerAddress=pekko://test/user/follower149, votingMember=false] 01:21:35,626 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:35,626 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server2 01:21:35,626 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:35,626 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server2: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:35,626 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): Number of entries to be appended = 1 01:21:35,626 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 01:21:35,626 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,627 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - new-server: Append log entry and persist JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:35,627 PM [new-server-%d990405] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-16008788317331156117/new-server/snapshot-0000000068B9925F-256115AA.tmp 01:21:35,627 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - new-server: Updated server config: isVoting: false, peers: [PeerInfo [id=leader, address=null, votingState=VOTING], PeerInfo [id=new-server2, address=null, votingState=NON_VOTING]] 01:21:35,627 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): Log size is now 2 01:21:35,627 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): Commit index set to 1 01:21:35,627 PM [test-pekko.actor.default-dispatcher-11] [ERROR] OneForOneStrategy - Cannot invoke "org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.getReplicatedToAllIndex()" because "currentBehavior" is null java.lang.NullPointerException: Cannot invoke "org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.getReplicatedToAllIndex()" because "currentBehavior" is null at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.SnapshotManager.trimLog(SnapshotManager.java:663) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.performSnapshotWithoutCapture(RaftActorBehavior.java:438) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.Follower.handleAppendEntries(Follower.java:222) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.appendEntries(RaftActorBehavior.java:152) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.handleMessage(RaftActorBehavior.java:375) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.Follower.handleMessage(Follower.java:502) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.ForwardMessageToBehaviorActor.onReceive(ForwardMessageToBehaviorActor.java:25) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.UntypedAbstractActor$$anon$1.applyOrElse(AbstractActor.scala:341) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:481) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:272) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:233) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:245) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188) 01:21:35,628 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=new-server, logLastIndex=1, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=true, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:35,628 PM [leader-%d613343] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-16008788317331156117/leader/snapshot-0000000068B9925F-25423887.v1 01:21:35,628 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 1 (of expected -1) 01:21:35,628 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 01:21:35,628 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - new-server (Follower): Setting last applied to 1 01:21:35,628 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from 0 to 1 01:21:35,628 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for new-server updated: matchIndex: 1, nextIndex: 2 01:21:35,628 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,628 PM [leader-%d613343] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-16008788317331156117/leader/snapshot-0000000068B9925F-03C792B9.v1 01:21:35,629 PM [leader-%d613343] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-16008788317331156117/leader/snapshot-0000000068B9925F-25423887.v1 01:21:35,629 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - new-server: awaiting more completions to resolve 1 synchronized callback(s) 01:21:35,629 PM [leader-%d613343] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16008788317331156117/leader/TermInfo.properties 01:21:35,629 PM [leader-%d613343] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16008788317331156117/leader/journal-v1-1.log 01:21:35,629 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:35,629 PM [leader-%d613343] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16008788317331156117/leader/metadata-v1-1.log 01:21:35,629 PM [leader-%d613343] [DEBUG] RaftStorage - leader: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.063410873Z, file=/tmp/junit-16008788317331156117/leader/snapshot-0000000068B9925F-03C792B9.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:35,629 PM [new-server-%d990405] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-16008788317331156117/new-server/snapshot-0000000068B9925F-256115AA.v1 01:21:35,629 PM [new-server-%d990405] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-16008788317331156117/new-server/snapshot-0000000068B9925F-04338DF2.v1 01:21:35,629 PM [new-server-%d990405] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-16008788317331156117/new-server/snapshot-0000000068B9925F-256115AA.v1 [ERROR] [09/04/2025 13:21:35.626] [test-pekko.actor.internal-dispatcher-4] [pekko://test/user/follower149] Cannot invoke "org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.getReplicatedToAllIndex()" because "currentBehavior" is null java.lang.NullPointerException: Cannot invoke "org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.getReplicatedToAllIndex()" because "currentBehavior" is null at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.SnapshotManager.trimLog(SnapshotManager.java:663) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.performSnapshotWithoutCapture(RaftActorBehavior.java:438) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.Follower.handleAppendEntries(Follower.java:222) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.appendEntries(RaftActorBehavior.java:152) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.RaftActorBehavior.handleMessage(RaftActorBehavior.java:375) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.behaviors.Follower.handleMessage(Follower.java:502) at org.opendaylight.controller.cluster.raft@12.0.0-SNAPSHOT/org.opendaylight.controller.cluster.raft.ForwardMessageToBehaviorActor.onReceive(ForwardMessageToBehaviorActor.java:25) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.UntypedAbstractActor$$anon$1.applyOrElse(AbstractActor.scala:341) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:481) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:272) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:233) at org.opendaylight.controller.repackaged.pekko@12.0.0-SNAPSHOT/org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:245) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188) 01:21:35,629 PM [new-server-%d990405] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16008788317331156117/new-server/TermInfo.properties 01:21:35,629 PM [new-server-%d990405] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16008788317331156117/new-server/journal-v1-1.log 01:21:35,630 PM [new-server-%d990405] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16008788317331156117/new-server/metadata-v1-1.log 01:21:35,630 PM [new-server-%d990405] [DEBUG] RaftStorage - new-server: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.070487538Z, file=/tmp/junit-16008788317331156117/new-server/snapshot-0000000068B9925F-04338DF2.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:35,630 PM [new-server-%d990405] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.627119530Z, file=/tmp/junit-16008788317331156117/new-server/snapshot-0000000068B9925F-256115AA.v1, size=100, entryCompress=NONE, stateCompress=NONE} 01:21:35,630 PM [new-server-%d990405] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:35,630 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:35,630 PM [leader-%d613343] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.625096839Z, file=/tmp/junit-16008788317331156117/leader/snapshot-0000000068B9925F-25423887.v1, size=100, entryCompress=NONE, stateCompress=NONE} 01:21:35,630 PM [leader-%d613343] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:35,630 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:35,630 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - new-server: persist complete JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:35,630 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:35,630 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - new-server: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=false], ServerInfo[peerId=new-server2, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@44a5c139} 01:21:35,630 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - new-server: synchronized callbacks after 1.597 ms 01:21:35,676 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServersAsNonVoting ending 01:21:35,677 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower149] 01:21:35,690 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector150#-823703607] 01:21:35,690 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server151] 01:21:35,691 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#305,new-server-writer-54]/waiting 01:21:35,691 PM [new-server-writer-54] [DEBUG] JournalWriteTask - new-server: received 1 commands after 669.5 ms 01:21:35,691 PM [new-server-writer-54] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 37.19 μs 01:21:35,691 PM [new-server-writer-54] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:35,692 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:35,692 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:35,692 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:35,692 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:35,692 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader152] 01:21:35,693 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#306,leader-writer-55]/waiting 01:21:35,693 PM [leader-writer-55] [DEBUG] JournalWriteTask - leader: received 1 commands after 658.8 ms 01:21:35,693 PM [leader-writer-55] [DEBUG] JournalWriteTask - leader: completed 1 commands in 17.52 μs 01:21:35,693 PM [leader-writer-55] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:35,693 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:35,693 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal closed 01:21:35,693 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:35,693 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:35,694 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leaderCollector153] 01:21:35,702 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithPriorSnapshotInProgress starting 01:21:35,703 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server156] [WARN] [09/04/2025 13:21:35.703] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server156]: org.opendaylight.controller.cluster.common.actor.Monitor@49eb9edf 01:21:35,705 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:35,705 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: started executor 01:21:35,705 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: started executor 01:21:35,706 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:35,707 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:35,707 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:35,707 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:35,707 PM [new-server-writer-56] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:35,707 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-13381191536517253525/new-server/TermInfo.properties does not exist 01:21:35,707 PM [new-server-writer-56] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:35,711 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:35,711 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:35,712 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-13381191536517253525/new-server/TermInfo.properties 01:21:35,712 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-13381191536517253525/new-server/journal-v1-1.log 01:21:35,712 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-13381191536517253525/new-server/metadata-v1-1.log 01:21:35,712 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:35,712 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:35,712 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - new-server: Recovery completed in in 4.659 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:35,712 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:35,712 PM [test-pekko.actor.default-dispatcher-13] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:35,713 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader157] [WARN] [09/04/2025 13:21:35.713] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader157]: org.opendaylight.controller.cluster.common.actor.Monitor@349b321 01:21:35,713 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:35,713 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:35,714 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:35,714 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:35,714 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:35,714 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:35,714 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:35,714 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:35,714 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-13381191536517253525/leader/TermInfo.properties does not exist 01:21:35,714 PM [leader-writer-57] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:35,714 PM [leader-writer-57] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:35,716 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:35,716 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:35,716 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/TermInfo.properties 01:21:35,716 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/journal-v1-1.log 01:21:35,716 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/metadata-v1-1.log 01:21:35,716 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: no eligible files found 01:21:35,717 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:35,717 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Recovery completed in in 2.305 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:35,717 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:35,717 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [] 01:21:35,718 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Take a snapshot of current state. lastReplicatedLog is null and replicatedToAllIndex is -1 01:21:35,718 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 01:21:35,718 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=true] 01:21:35,718 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: captured snapshot at lastSequenceNumber: 0 01:21:35,718 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Persising snapshot at EntryInfo[index=-1, term=-1]/EntryInfo[index=-1, term=-1] 01:21:35,718 PM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - leader: support() called 01:21:35,719 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: user triggered or root overwrite snapshot encountered, trimming log up to last applied index -1 01:21:35,719 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 01:21:35,719 PM [leader-%d396067] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-2ADF7163.tmp 01:21:35,723 PM [leader-%d396067] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-2ADF7163.v1 01:21:35,723 PM [leader-%d396067] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/TermInfo.properties 01:21:35,723 PM [leader-%d396067] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/journal-v1-1.log 01:21:35,723 PM [leader-%d396067] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/metadata-v1-1.log 01:21:35,723 PM [leader-%d396067] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-2ADF7163.v1 01:21:35,724 PM [leader-%d396067] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.719286627Z, file=/tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-2ADF7163.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:35,724 PM [leader-%d396067] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:35,724 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server156, votingMember=true], state: Idle 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server156, votingMember=true] 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: captureToInstall should not be called in state PersistCapture[lastJournalIndex=0, request=CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=true]] 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Snapshot already in progress - waiting for completion 01:21:35,831 PM [main] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: snapshot is durable as of 2025-09-04T13:21:35.719286627Z 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: Snapshot success, discarding journal entries up to 0 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onSnapshotComplete 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on new-server 01:21:35,831 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: last applied journal index prior to capture: 0 01:21:35,832 PM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - leader: support() called 01:21:35,832 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating capture snapshot for new server new-server 01:21:35,832 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorSnapshotMessageSupport - leader: SnapshotComplete received 01:21:35,832 PM [leader-%d396068] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:35,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:35,833 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Persising snapshot at EntryInfo[index=-1, term=-1]/EntryInfo[index=-1, term=-1] 01:21:35,833 PM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - leader: support() called 01:21:35,833 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 01:21:35,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): sendInstallSnapshot 01:21:35,833 PM [leader-%d396069] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-31ACD7A4.tmp 01:21:35,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Snapshot 157 bytes, total chunks to send: 1 01:21:35,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Next chunk: total length=157, offset=0, size=157, hashCode=1546036366 01:21:35,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): next snapshot chunk size for follower new-server: 157 01:21:35,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): InstallSnapshot sent to follower Vector(user, new-server156), Chunk: 1/1 01:21:35,833 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - new-server (Follower): Term 1 in "InstallSnapshot{term=1, leaderId=leader, lastIncludedIndex=-1, lastIncludedTerm=-1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1]}" message is greater than follower's term 0 - updating term 01:21:35,836 PM [leader-%d396069] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-31ACD7A4.v1 01:21:35,836 PM [leader-%d396069] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-31ACD7A4.v1 01:21:35,836 PM [leader-%d396069] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/TermInfo.properties 01:21:35,836 PM [leader-%d396069] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/journal-v1-1.log 01:21:35,836 PM [leader-%d396069] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/metadata-v1-1.log 01:21:35,836 PM [leader-%d396069] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-2ADF7163.v1 01:21:35,836 PM [leader-%d396069] [DEBUG] RaftStorage - leader: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.719286627Z, file=/tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-2ADF7163.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:35,836 PM [leader-%d396069] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.833410980Z, file=/tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-31ACD7A4.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:35,836 PM [leader-%d396069] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:35,837 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=1, votedFor=null 01:21:35,837 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:35,837 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Follower - new-server (Follower): handleInstallSnapshot: InstallSnapshot{term=1, leaderId=leader, lastIncludedIndex=-1, lastIncludedTerm=-1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1]} 01:21:35,837 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: snapshot is durable as of 2025-09-04T13:21:35.833410980Z 01:21:35,837 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - new-server: Last sync leader does not match current leader leader, need to catch up to -1 01:21:35,837 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: Snapshot success, discarding journal entries up to 0 01:21:35,837 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotTracker - new-server (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 01:21:35,837 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorSnapshotMessageSupport - leader: SnapshotComplete received 01:21:35,837 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - new-server (Follower): Snapshot received from leader: leader 01:21:35,837 PM [test-pekko.actor.default-dispatcher-11] [INFO] SnapshotManager - new-server: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=157}} 01:21:35,837 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - new-server: support() called [WARN] [09/04/2025 13:21:35.837] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/new-server156] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=false] 01:21:35,838 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - new-server: Converted InstallSnapshot from leader: leader to state 01:21:35,838 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SnapshotManager - new-server: last applied journal index prior to persisting applied snapshot: 0 01:21:35,838 PM [test-pekko.actor.default-dispatcher-11] [INFO] MockRaftActor - new-server: support() called 01:21:35,839 PM [new-server-%d377016] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-13381191536517253525/new-server/snapshot-0000000068B9925F-3202230E.tmp 01:21:35,844 PM [new-server-%d377016] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-13381191536517253525/new-server/snapshot-0000000068B9925F-3202230E.v1 01:21:35,844 PM [new-server-%d377016] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-13381191536517253525/new-server/snapshot-0000000068B9925F-3202230E.v1 01:21:35,844 PM [new-server-%d377016] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-13381191536517253525/new-server/TermInfo.properties 01:21:35,844 PM [new-server-%d377016] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-13381191536517253525/new-server/journal-v1-1.log 01:21:35,844 PM [new-server-%d377016] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-13381191536517253525/new-server/metadata-v1-1.log 01:21:35,844 PM [new-server-%d377016] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.839000846Z, file=/tmp/junit-13381191536517253525/new-server/snapshot-0000000068B9925F-3202230E.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:35,844 PM [new-server-%d377016] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:35,844 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:35,844 PM [test-pekko.actor.default-dispatcher-5] [INFO] SnapshotManager - new-server: snapshot is durable as of 2025-09-04T13:21:35.839000846Z 01:21:35,844 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - new-server: support() called 01:21:35,845 PM [test-pekko.actor.default-dispatcher-5] [INFO] MockRaftActor - new-server: applySnapshot called 01:21:35,845 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - new-server (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=1, followerId=new-server, chunkIndex=1, success=true} 01:21:35,845 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SnapshotManager - new-server: Snapshot success, discarding journal entries up to 0 01:21:35,845 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorSnapshotMessageSupport - new-server: SnapshotComplete received 01:21:35,845 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=1, followerId=new-server, chunkIndex=1, success=true} 01:21:35,845 PM [test-pekko.actor.default-dispatcher-11] [INFO] AbstractLeader - leader (Leader): Snapshot successfully installed on follower new-server (last chunk 1) - matchIndex set to -1, nextIndex set to 0 01:21:35,845 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - Sent message UnInitializedFollowerSnapshotReply to self 01:21:35,845 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: onUnInitializedFollowerSnapshotReply: UnInitializedFollowerSnapshotReply [followerId=new-server] 01:21:35,845 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]] 01:21:35,845 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: Persist data index=0 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,845 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:35,846 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=4f954781-3a34-42ba-8abc-cccf4bb51072}, logIndex: 0, isSendImmediate: true 01:21:35,846 PM [leader-%d396070] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-326BE6C2.tmp 01:21:35,846 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:35,846 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,846 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower new-server 01:21:35,846 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:35,846 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server156, votingMember=true] 01:21:35,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:35,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): Number of entries to be appended = 1 01:21:35,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:21:35,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - new-server: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:35,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - new-server: Updated server config: isVoting: true, peers: [PeerInfo [id=leader, address=null, votingState=VOTING]] 01:21:35,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): Log size is now 1 01:21:35,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - new-server (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=new-server, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=true, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:35,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached -1 (of expected -1) 01:21:35,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - new-server: awaiting more completions to resolve 1 synchronized callback(s) 01:21:35,846 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for new-server updated: matchIndex: 0, nextIndex: 1 01:21:35,846 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,846 PM [new-server-%d377017] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-13381191536517253525/new-server/snapshot-0000000068B9925F-3275A7D8.tmp 01:21:35,849 PM [leader-%d396070] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-326BE6C2.v1 01:21:35,849 PM [leader-%d396070] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-31ACD7A4.v1 01:21:35,849 PM [leader-%d396070] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/TermInfo.properties 01:21:35,849 PM [leader-%d396070] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/journal-v1-1.log 01:21:35,849 PM [leader-%d396070] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-13381191536517253525/leader/metadata-v1-1.log 01:21:35,849 PM [leader-%d396070] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-326BE6C2.v1 01:21:35,849 PM [leader-%d396070] [DEBUG] RaftStorage - leader: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.833410980Z, file=/tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-31ACD7A4.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:35,849 PM [leader-%d396070] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.845932226Z, file=/tmp/junit-13381191536517253525/leader/snapshot-0000000068B9925F-326BE6C2.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:35,850 PM [leader-%d396070] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:35,850 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:35,850 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:35,850 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:21:35,850 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 0 01:21:35,850 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to 0 01:21:35,850 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:35,850 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:35,851 PM [new-server-%d377017] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-13381191536517253525/new-server/snapshot-0000000068B9925F-3275A7D8.v1 01:21:35,851 PM [new-server-%d377017] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-13381191536517253525/new-server/snapshot-0000000068B9925F-3275A7D8.v1 01:21:35,851 PM [new-server-%d377017] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-13381191536517253525/new-server/snapshot-0000000068B9925F-3202230E.v1 01:21:35,851 PM [new-server-%d377017] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-13381191536517253525/new-server/TermInfo.properties 01:21:35,851 PM [new-server-%d377017] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-13381191536517253525/new-server/journal-v1-1.log 01:21:35,852 PM [new-server-%d377017] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-13381191536517253525/new-server/metadata-v1-1.log 01:21:35,852 PM [new-server-%d377017] [DEBUG] RaftStorage - new-server: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.839000846Z, file=/tmp/junit-13381191536517253525/new-server/snapshot-0000000068B9925F-3202230E.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:35,852 PM [new-server-%d377017] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.846571480Z, file=/tmp/junit-13381191536517253525/new-server/snapshot-0000000068B9925F-3275A7D8.v1, size=86, entryCompress=NONE, stateCompress=NONE} 01:21:35,852 PM [new-server-%d377017] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:35,852 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:35,852 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - new-server: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:35,852 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - new-server: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@6517a230} 01:21:35,852 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - new-server: synchronized callbacks after 5.803 ms [WARN] [09/04/2025 13:21:35.852] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/new-server156] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=true] 01:21:35,897 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithPriorSnapshotInProgress ending 01:21:35,897 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower154] 01:21:35,909 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector155#723157842] 01:21:35,909 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server156] 01:21:35,910 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#311,new-server-writer-56]/waiting 01:21:35,910 PM [new-server-writer-56] [DEBUG] JournalWriteTask - new-server: received 1 commands after 202.8 ms 01:21:35,910 PM [new-server-writer-56] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 84.16 μs 01:21:35,910 PM [new-server-writer-56] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:35,911 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:35,911 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:35,911 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:35,911 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:35,911 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader157] 01:21:35,911 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#312,leader-writer-57]/waiting 01:21:35,912 PM [leader-writer-57] [DEBUG] JournalWriteTask - leader: received 1 commands after 197.3 ms 01:21:35,912 PM [leader-writer-57] [DEBUG] JournalWriteTask - leader: completed 1 commands in 25.42 μs 01:21:35,912 PM [leader-writer-57] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:35,912 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:35,912 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal closed 01:21:35,912 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:35,912 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:35,912 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leaderCollector158] 01:21:35,920 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeLeaderToNonVotingInSingleNode starting 01:21:35,921 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader160] [WARN] [09/04/2025 13:21:35.921] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader160]: org.opendaylight.controller.cluster.common.actor.Monitor@5bf381a1 01:21:35,922 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:35,922 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:35,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:35,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:35,923 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:35,924 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:35,924 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:35,924 PM [leader-writer-58] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:35,924 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:35,924 PM [leader-writer-58] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:35,924 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-14401419763252857780/leader/TermInfo.properties does not exist 01:21:35,927 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:35,927 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:35,927 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-14401419763252857780/leader/TermInfo.properties 01:21:35,928 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-14401419763252857780/leader/journal-v1-1.log 01:21:35,928 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-14401419763252857780/leader/metadata-v1-1.log 01:21:35,928 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: no eligible files found 01:21:35,928 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:35,928 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Recovery completed in in 3.199 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:35,928 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:35,928 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [] 01:21:35,939 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={leader=false}, serversVisited=[]], state: Idle 01:21:35,939 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:35,939 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Returning INVALID_REQUEST for operation ChangeServersVotingStatus [serverVotingStatusMap={leader=false}, serversVisited=[]] 01:21:35,939 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeLeaderToNonVotingInSingleNode ending 01:21:35,940 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower159] 01:21:35,951 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader160] 01:21:35,951 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#320,leader-writer-58]/waiting 01:21:35,951 PM [leader-writer-58] [DEBUG] JournalWriteTask - leader: received 1 commands after 26.72 ms 01:21:35,951 PM [leader-writer-58] [DEBUG] JournalWriteTask - leader: completed 1 commands in 27.33 μs 01:21:35,951 PM [leader-writer-58] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:35,952 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:35,952 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal closed 01:21:35,952 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:35,952 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:35,956 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeLeaderToNonVoting starting 01:21:35,957 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader164] [WARN] [09/04/2025 13:21:35.957] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader164]: org.opendaylight.controller.cluster.common.actor.Monitor@6fdf5e86 01:21:35,958 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:35,958 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:35,958 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:35,958 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:35,959 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:35,961 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:35,961 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:35,961 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:35,961 PM [leader-writer-59] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:35,961 PM [leader-writer-59] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:35,961 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-5660662718287255979/leader/TermInfo.properties does not exist 01:21:35,963 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:35,963 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:35,964 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5660662718287255979/leader/TermInfo.properties 01:21:35,964 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5660662718287255979/leader/journal-v1-1.log 01:21:35,964 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5660662718287255979/leader/metadata-v1-1.log 01:21:35,964 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: no eligible files found 01:21:35,964 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:35,964 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Recovery completed in in 2.746 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:35,964 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:35,964 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower2, follower] 01:21:35,964 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:21:35,964 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [09/04/2025 13:21:35.964] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower2163] received dead letter from TestActor[pekko://test/user/leader164]: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} [WARN] [09/04/2025 13:21:35.965] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/follower162] received dead letter from TestActor[pekko://test/user/leader164]: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 01:21:35,966 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/follower162] [WARN] [09/04/2025 13:21:35.967] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower162]: org.opendaylight.controller.cluster.common.actor.Monitor@531ac4f3 01:21:35,968 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower: Starting recovery with journal batch size 1000 01:21:35,968 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: started executor 01:21:35,968 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: started executor 01:21:35,969 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower: metadata journal open: replayFrom=1 applyTo=0 01:21:35,969 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:35,969 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - follower: journal open: applyTo=0 01:21:35,970 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower: recovering to non-persistent 01:21:35,970 PM [follower-writer-60] [DEBUG] JournalWriteTask - follower: journal writer started 01:21:35,970 PM [follower-writer-60] [DEBUG] JournalWriteTask - follower: waiting to receive commands 01:21:35,970 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/junit-5660662718287255979/follower/TermInfo.properties does not exist 01:21:35,972 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=0, votedFor=null 01:21:35,972 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - follower: Local TermInfo store seeded with TermInfo{term=0} 01:21:35,972 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-5660662718287255979/follower/TermInfo.properties 01:21:35,972 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-5660662718287255979/follower/journal-v1-1.log 01:21:35,972 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-5660662718287255979/follower/metadata-v1-1.log 01:21:35,973 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower: no eligible files found 01:21:35,973 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from -1 to -1 01:21:35,973 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - follower: Recovery completed in in 2.905 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:35,973 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower: Recovery completed and can restore from snapshot 01:21:35,974 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/follower2163] [WARN] [09/04/2025 13:21:35.974] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower2163]: org.opendaylight.controller.cluster.common.actor.Monitor@496ab72 01:21:35,975 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActor - follower2: Starting recovery with journal batch size 1000 01:21:35,975 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower2: started executor 01:21:35,975 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower2: started executor 01:21:35,976 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower2: metadata journal open: replayFrom=1 applyTo=0 01:21:35,976 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] EntryJournalV1 - follower2: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:35,976 PM [test-pekko.actor.default-dispatcher-11] [INFO] EnabledRaftStorage - follower2: journal open: applyTo=0 01:21:35,976 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower2: recovering to non-persistent 01:21:35,976 PM [follower2-writer-61] [DEBUG] JournalWriteTask - follower2: journal writer started 01:21:35,976 PM [follower2-writer-61] [DEBUG] JournalWriteTask - follower2: waiting to receive commands 01:21:35,976 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - /tmp/junit-5660662718287255979/follower2/TermInfo.properties does not exist 01:21:35,979 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - follower2: Set currentTerm=0, votedFor=null 01:21:35,979 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - follower2: Local TermInfo store seeded with TermInfo{term=0} 01:21:35,979 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-5660662718287255979/follower2/TermInfo.properties 01:21:35,979 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-5660662718287255979/follower2/journal-v1-1.log 01:21:35,979 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-5660662718287255979/follower2/metadata-v1-1.log 01:21:35,979 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftStorage - follower2: no eligible files found 01:21:35,979 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - follower2: Moving last applied index from -1 to -1 01:21:35,979 PM [test-pekko.actor.default-dispatcher-11] [INFO] Recovery - follower2: Recovery completed in in 2.352 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:35,979 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - follower2: Recovery completed and can restore from snapshot 01:21:35,989 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={leader=false}, serversVisited=[]], state: Idle 01:21:35,990 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:35,990 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - leader: Updated server config: isVoting: false, peers: [PeerInfo [id=follower2, address=pekko://test/user/follower2163, votingState=VOTING], PeerInfo [id=follower, address=pekko://test/user/follower162, votingState=VOTING]] 01:21:35,990 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]] 01:21:35,990 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: Persist data index=0 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]} 01:21:35,990 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=true} 01:21:35,990 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=9083ce10-98e7-422e-9d9f-f0904087a705}, logIndex: 0, isSendImmediate: true 01:21:35,990 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:35,990 PM [leader-%d352065] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-5660662718287255979/leader/snapshot-0000000068B9925F-3B0A4C9A.tmp 01:21:35,990 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,990 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,990 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower2 (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:35,991 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation ChangeServersVotingStatus [serverVotingStatusMap={leader=false}, serversVisited=[]] 01:21:35,991 PM [test-pekko.actor.default-dispatcher-5] [INFO] Follower - follower (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:35,993 PM [leader-%d352065] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-5660662718287255979/leader/snapshot-0000000068B9925F-3B0A4C9A.v1 01:21:35,993 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower2: Set currentTerm=1, votedFor=null 01:21:35,993 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower2: Last sync leader does not match current leader leader, need to catch up to -1 01:21:35,993 PM [leader-%d352065] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-5660662718287255979/leader/snapshot-0000000068B9925F-3B0A4C9A.v1 01:21:35,993 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower2 updated: matchIndex: -1, nextIndex: 0 [WARN] [09/04/2025 13:21:35.993] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower2163] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower2, initialSyncDone=false] 01:21:35,993 PM [leader-%d352065] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5660662718287255979/leader/TermInfo.properties 01:21:35,993 PM [leader-%d352065] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5660662718287255979/leader/journal-v1-1.log 01:21:35,993 PM [leader-%d352065] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-5660662718287255979/leader/metadata-v1-1.log 01:21:35,993 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,994 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower follower2 01:21:35,994 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower follower2: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=true}]} 01:21:35,994 PM [leader-%d352065] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.990530714Z, file=/tmp/junit-5660662718287255979/leader/snapshot-0000000068B9925F-3B0A4C9A.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:35,994 PM [leader-%d352065] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:35,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=true}]} 01:21:35,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): Number of entries to be appended = 1 01:21:35,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:21:35,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]} 01:21:35,994 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:35,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower2: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=true} 01:21:35,994 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=true} 01:21:35,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - follower2: Updated server config: isVoting: true, peers: [PeerInfo [id=leader, address=pekko://test/user/leader164, votingState=NON_VOTING], PeerInfo [id=follower, address=pekko://test/user/follower162, votingState=VOTING]] 01:21:35,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): Log size is now 1 01:21:35,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=follower2, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:35,994 PM [follower2-%d652628] [DEBUG] RaftStorage - follower2: starting snapshot writeout to /tmp/junit-5660662718287255979/follower2/snapshot-0000000068B9925F-3B44D66D.tmp 01:21:35,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower2: Lagging 0 entries behind leader leader and reached -1 (of expected -1) 01:21:35,994 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower2: awaiting more completions to resolve 1 synchronized callback(s) 01:21:35,994 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=1, votedFor=null 01:21:35,994 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower: Last sync leader does not match current leader leader, need to catch up to -1 01:21:35,994 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower2 updated: matchIndex: 0, nextIndex: 1 01:21:35,994 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:21:35,994 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 0 01:21:35,994 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to 0 01:21:35,994 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]} 01:21:35,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 [WARN] [09/04/2025 13:21:35.995] [test-pekko.actor.default-dispatcher-5] [pekko://test/user/follower162] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower, initialSyncDone=false] 01:21:35,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: -1, nextIndex: 0 01:21:35,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower follower 01:21:35,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower follower: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=false}]} 01:21:35,995 PM [test-pekko.actor.default-dispatcher-11] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]} has been successfully replicated to a majority of followers 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=false}]} 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower (Follower): Number of entries to be appended = 1 01:21:35,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - Leader changed to non-voting - trying leadership transfer 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]} 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=false} 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PeerInfos - follower: Updated server config: isVoting: true, peers: [PeerInfo [id=leader, address=pekko://test/user/leader164, votingState=NON_VOTING], PeerInfo [id=follower2, address=pekko://test/user/follower2163, votingState=VOTING]] 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower (Follower): Log size is now 1 01:21:35,995 PM [follower-%d618244] [DEBUG] RaftStorage - follower: starting snapshot writeout to /tmp/junit-5660662718287255979/follower/snapshot-0000000068B9925F-3B5755C8.tmp 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower (Follower): Commit index set to 0 01:21:35,995 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActor - leader: Initiating leader transfer 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=follower, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActorBehavior - follower (Follower): Setting last applied to 0 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from -1 to 0 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]} 01:21:35,995 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower: awaiting more completions to resolve 1 synchronized callback(s) 01:21:35,996 PM [follower2-%d652628] [DEBUG] RaftStorage - follower2: finished snapshot writeout to /tmp/junit-5660662718287255979/follower2/snapshot-0000000068B9925F-3B44D66D.v1 01:21:35,997 PM [follower2-%d652628] [DEBUG] RaftStorage - follower2: selected SNAPSHOT_V1 to handle file /tmp/junit-5660662718287255979/follower2/snapshot-0000000068B9925F-3B44D66D.v1 01:21:35,997 PM [follower2-%d652628] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-5660662718287255979/follower2/TermInfo.properties 01:21:35,997 PM [follower2-%d652628] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-5660662718287255979/follower2/journal-v1-1.log 01:21:35,997 PM [follower2-%d652628] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-5660662718287255979/follower2/metadata-v1-1.log 01:21:35,997 PM [follower2-%d652628] [DEBUG] RaftStorage - follower2: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.994367085Z, file=/tmp/junit-5660662718287255979/follower2/snapshot-0000000068B9925F-3B44D66D.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:35,997 PM [follower2-%d652628] [DEBUG] RaftStorageCompleter - follower2: 1 completion(s) pending 01:21:35,997 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower2: running 1 completion(s) 01:21:35,997 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower2: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=false} 01:21:35,997 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower2: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@49797156} 01:21:35,997 PM [follower-%d618244] [DEBUG] RaftStorage - follower: finished snapshot writeout to /tmp/junit-5660662718287255979/follower/snapshot-0000000068B9925F-3B5755C8.v1 01:21:35,997 PM [follower-%d618244] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-5660662718287255979/follower/TermInfo.properties 01:21:35,997 PM [follower-%d618244] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-5660662718287255979/follower/journal-v1-1.log 01:21:35,997 PM [follower-%d618244] [DEBUG] RaftStorage - follower: selected SNAPSHOT_V1 to handle file /tmp/junit-5660662718287255979/follower/snapshot-0000000068B9925F-3B5755C8.v1 01:21:35,997 PM [follower-%d618244] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-5660662718287255979/follower/metadata-v1-1.log 01:21:35,997 PM [follower-%d618244] [DEBUG] RaftStorage - follower: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:35.995579336Z, file=/tmp/junit-5660662718287255979/follower/snapshot-0000000068B9925F-3B5755C8.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:35,997 PM [follower-%d618244] [DEBUG] RaftStorageCompleter - follower: 1 completion(s) pending 01:21:35,997 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower: running 1 completion(s) 01:21:35,997 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] ReplicatedLogImpl - follower: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=false} 01:21:35,997 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower2: synchronized callbacks after 2.761 ms 01:21:35,997 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@40c65f11} 01:21:35,997 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): Commit index set to 0 01:21:35,997 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftStorageCompleter - follower: synchronized callbacks after 2.043 ms 01:21:35,997 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower2: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:35,997 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:21:35,997 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower2 (Follower): Setting last applied to 0 01:21:35,997 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower2: Moving last applied index from -1 to 0 01:21:35,997 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower2: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=false]]} [WARN] [09/04/2025 13:21:35.998] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower162] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower, initialSyncDone=true] [WARN] [09/04/2025 13:21:35.998] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower2163] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower2, initialSyncDone=true] 01:21:35,998 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower2: onLeaderTransitioning: LeaderTransitioning [leaderId=leader] 01:21:35,998 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] RaftActor - follower: onLeaderTransitioning: LeaderTransitioning [leaderId=leader] 01:21:35,998 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader: pauseLeader successfully completed - doing transfer 01:21:35,998 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Leader - leader (Leader): Attempting to transfer leadership 01:21:35,998 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,998 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,998 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower follower 01:21:35,998 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower2: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:35,998 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 0, nextIndex: 1 01:21:35,999 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,999 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Leader - leader (Leader): tryToCompleteLeadershipTransfer: followerId: follower, matchIndex: 0, lastIndex: 0, isVoting: true 01:21:35,999 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Leader - leader (Leader): Follower's log matches - sending ElectionTimeout 01:21:35,999 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,999 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:35,999 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower2: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:35,999 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] Leader - leader (Leader): Leader transfer complete 01:21:35,999 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorLeadershipTransferCohort - leader: leader transfer complete - waiting for new leader 01:21:35,999 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:35,999 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:35,999 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Follower - follower (Follower): Received TimeoutNow - switching to Candidate 01:21:35,999 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower (Candidate): Election: Candidate has following voting peers: [follower2] 01:21:36,1 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=2, votedFor=follower 01:21:36,1 PM [test-pekko.actor.default-dispatcher-5] [INFO] Candidate - follower (Candidate): Starting new election term 2 01:21:36,1 PM [test-pekko.actor.default-dispatcher-5] [DEBUG] Candidate - follower (Candidate): Sending RequestVote{term=2, candidateId=follower, lastLogIndex=0, lastLogTerm=1} to peer follower2 01:21:36,1 PM [test-pekko.actor.default-dispatcher-5] [INFO] RaftActorBehavior - follower (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:21:36,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower2 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:21:36,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower2 (Follower): Cluster state: [] 01:21:36,1 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower2 (Follower): Candidate in requestVote:RequestVote{term=2, candidateId=follower, lastLogIndex=0, lastLogTerm=1} with higher term appears reachable, updating term. 01:21:36,1 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower2 (Follower): Term 2 in "RequestVote{term=2, candidateId=follower, lastLogIndex=0, lastLogTerm=1}" message is greater than follower's term 1 - updating term 01:21:36,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower2: Set currentTerm=2, votedFor=null 01:21:36,3 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower2 (Follower): In requestVote: RequestVote{term=2, candidateId=follower, lastLogIndex=0, lastLogTerm=1} - currentTerm: 2, votedFor: null, lastIndex: 0, lastTerm: 1 01:21:36,4 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower2: Set currentTerm=2, votedFor=follower 01:21:36,4 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower2 (Follower): requestVote returning: RequestVoteReply{term=2, voteGranted=true} 01:21:36,5 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - follower (Candidate): RaftRPC message received RequestVoteReply{term=2, voteGranted=true}, my term is 2 01:21:36,5 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] Candidate - follower (Candidate): handleRequestVoteReply: RequestVoteReply{term=2, voteGranted=true}, current voteCount: 1 01:21:36,5 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - follower (Leader): Election: Leader has following peers: [leader, follower2] 01:21:36,5 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - follower (Leader): Checking sendAppendEntries for follower leader: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,5 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - follower (Leader): Checking sendAppendEntries for follower follower2: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,5 PM [test-pekko.actor.default-dispatcher-6] [INFO] RaftActorBehavior - follower (Candidate) :- Switching from behavior Candidate to Leader, election term: 2 01:21:36,5 PM [test-pekko.actor.default-dispatcher-14] [DEBUG] SyncStatusTracker - follower2: Last sync leader does not match current leader follower, need to catch up to -1 01:21:36,5 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - follower (Leader): handleAppendEntriesReply - FollowerLogInformation for follower2 updated: matchIndex: 0, nextIndex: 1 01:21:36,5 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] AbstractLeader - follower (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,5 PM [test-pekko.actor.default-dispatcher-6] [DEBUG] SyncStatusTracker - follower2: Lagging 0 entries behind leader follower and reached 0 (of expected -1) [WARN] [09/04/2025 13:21:36.005] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower2163] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower2, initialSyncDone=false] [WARN] [09/04/2025 13:21:36.005] [test-pekko.actor.default-dispatcher-14] [pekko://test/user/follower2163] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower2, initialSyncDone=true] 01:21:36,5 PM [test-pekko.actor.default-dispatcher-15] [INFO] AbstractLeader - leader (Leader): Term 2 in "AppendEntries{term=2, leaderId=follower, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than leader's term 1 - switching to Follower 01:21:36,7 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=2, votedFor=null 01:21:36,7 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActorBehavior - leader (Leader) :- Switching from behavior Leader to Follower, election term: 2 01:21:36,523 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower2: Lagging 0 entries behind leader follower and reached 0 (of expected -1) 01:21:36,523 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - leader: Last sync leader does not match current leader follower, need to catch up to -1 01:21:36,523 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorLeadershipTransferCohort - leader: leader changed to follower 01:21:36,523 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - follower (Leader): handleAppendEntriesReply - FollowerLogInformation for leader updated: matchIndex: 0, nextIndex: 1 01:21:36,523 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - follower (Leader): Checking sendAppendEntries for follower leader: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,523 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorLeadershipTransferCohort - leader: Successfully transferred leadership to follower in 527.5 ms 01:21:36,523 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader: leader transfer succeeded after change to non-voting 01:21:36,524 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - leader: Lagging 0 entries behind leader follower and reached 0 (of expected -1) [WARN] [09/04/2025 13:21:36.524] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/leader164] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader, initialSyncDone=false] [WARN] [09/04/2025 13:21:36.524] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/leader164] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=leader, initialSyncDone=true] 01:21:36,545 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeLeaderToNonVoting ending 01:21:36,545 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower161] 01:21:36,557 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader164] 01:21:36,557 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#321,leader-writer-59]/waiting 01:21:36,557 PM [leader-writer-59] [DEBUG] JournalWriteTask - leader: received 1 commands after 596.2 ms 01:21:36,557 PM [leader-writer-59] [DEBUG] JournalWriteTask - leader: completed 1 commands in 31.43 μs 01:21:36,557 PM [leader-writer-59] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:36,558 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:36,558 PM [test-pekko.actor.default-dispatcher-15] [INFO] EnabledRaftStorage - leader: journal closed 01:21:36,558 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:36,558 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:36,558 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leaderCollector165] 01:21:36,559 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector166#1895900745] 01:21:36,559 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower162] 01:21:36,559 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EnabledRaftStorage - follower: terminating thread VirtualThread[#322,follower-writer-60]/waiting 01:21:36,560 PM [follower-writer-60] [DEBUG] JournalWriteTask - follower: received 1 commands after 589.9 ms 01:21:36,560 PM [follower-writer-60] [DEBUG] JournalWriteTask - follower: completed 1 commands in 20.36 μs 01:21:36,560 PM [follower-writer-60] [DEBUG] JournalWriteTask - follower: journal writer stopped 01:21:36,560 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower: journal closed 01:21:36,560 PM [test-pekko.actor.default-dispatcher-15] [INFO] EnabledRaftStorage - follower: journal closed 01:21:36,560 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:36,560 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:36,560 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector167#-725004181] 01:21:36,561 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower2163] 01:21:36,561 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EnabledRaftStorage - follower2: terminating thread VirtualThread[#323,follower2-writer-61]/waiting 01:21:36,561 PM [follower2-writer-61] [DEBUG] JournalWriteTask - follower2: received 1 commands after 584.6 ms 01:21:36,561 PM [follower2-writer-61] [DEBUG] JournalWriteTask - follower2: completed 1 commands in 19.19 μs 01:21:36,561 PM [follower2-writer-61] [DEBUG] JournalWriteTask - follower2: journal writer stopped 01:21:36,561 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - follower2: journal closed 01:21:36,561 PM [test-pekko.actor.default-dispatcher-15] [INFO] EnabledRaftStorage - follower2: journal closed 01:21:36,561 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower2: stopped executor with 0 remaining tasks 01:21:36,561 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - follower2: stopped executor with 0 remaining tasks 01:21:36,567 PM [main] [INFO] RaftActorVotingConfigSupportTest - testOnApplyState starting 01:21:36,567 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] MockRaftActor - Actor created TestActor[pekko://test/user/leader169] [WARN] [09/04/2025 13:21:36.568] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader169]: org.opendaylight.controller.cluster.common.actor.Monitor@1ddfb6fd 01:21:36,569 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:36,569 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: started executor 01:21:36,569 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: started executor 01:21:36,570 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:36,571 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,571 PM [test-pekko.actor.default-dispatcher-15] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:36,571 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:36,571 PM [leader-writer-62] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:36,571 PM [leader-writer-62] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:36,571 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/junit-12705014464439003075/leader/TermInfo.properties does not exist 01:21:36,573 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=0, votedFor=null 01:21:36,573 PM [test-pekko.actor.default-dispatcher-15] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=0} 01:21:36,573 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12705014464439003075/leader/TermInfo.properties 01:21:36,573 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12705014464439003075/leader/journal-v1-1.log 01:21:36,573 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-12705014464439003075/leader/metadata-v1-1.log 01:21:36,573 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: no eligible files found 01:21:36,573 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:36,574 PM [test-pekko.actor.default-dispatcher-15] [INFO] Recovery - leader: Recovery completed in in 2.740 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,574 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:36,574 PM [main] [INFO] RaftActorVotingConfigSupportTest - testOnApplyState ending 01:21:36,574 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower168] 01:21:36,585 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader169] 01:21:36,585 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#329,leader-writer-62]/waiting 01:21:36,586 PM [leader-writer-62] [DEBUG] JournalWriteTask - leader: received 1 commands after 14.78 ms 01:21:36,586 PM [leader-writer-62] [DEBUG] JournalWriteTask - leader: completed 1 commands in 29.17 μs 01:21:36,586 PM [leader-writer-62] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:36,586 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:36,586 PM [test-pekko.actor.default-dispatcher-15] [INFO] EnabledRaftStorage - leader: journal closed 01:21:36,586 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:36,586 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:36,590 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServer starting 01:21:36,591 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader173] [WARN] [09/04/2025 13:21:36.591] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader173]: org.opendaylight.controller.cluster.common.actor.Monitor@58dd2b22 01:21:36,592 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:36,592 PM [test-pekko.actor.default-dispatcher-15] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:36,592 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: started executor 01:21:36,592 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: started executor 01:21:36,593 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:36,593 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,593 PM [test-pekko.actor.default-dispatcher-15] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:36,593 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:36,593 PM [leader-writer-63] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:36,593 PM [leader-writer-63] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:36,594 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - /tmp/junit-8046194582193304322/leader/TermInfo.properties does not exist 01:21:36,596 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:36,596 PM [test-pekko.actor.default-dispatcher-15] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:36,596 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-8046194582193304322/leader/TermInfo.properties 01:21:36,596 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-8046194582193304322/leader/journal-v1-1.log 01:21:36,596 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-8046194582193304322/leader/metadata-v1-1.log 01:21:36,596 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorage - leader: no eligible files found 01:21:36,596 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:36,596 PM [test-pekko.actor.default-dispatcher-15] [INFO] Recovery - leader: Recovery completed in in 2.636 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,596 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:36,596 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower2, follower, downNode] 01:21:36,597 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:21:36,597 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [09/04/2025 13:21:36.597] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower2172] received dead letter from TestActor[pekko://test/user/leader173]: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 01:21:36,597 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower downNode: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [09/04/2025 13:21:36.597] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/follower171] received dead letter from TestActor[pekko://test/user/leader173]: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} [WARN] [09/04/2025 13:21:36.597] [test-pekko.actor.default-dispatcher-12] [pekko://test/] unhandled message from TestActor[pekko://test/user/leader173]: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 01:21:36,599 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/follower171] [WARN] [09/04/2025 13:21:36.599] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower171]: org.opendaylight.controller.cluster.common.actor.Monitor@2ea61618 01:21:36,600 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower: Starting recovery with journal batch size 1000 01:21:36,600 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: started executor 01:21:36,600 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: started executor 01:21:36,601 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower: metadata journal open: replayFrom=1 applyTo=0 01:21:36,601 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,601 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - follower: journal open: applyTo=0 01:21:36,602 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower: recovering to non-persistent 01:21:36,602 PM [follower-writer-64] [DEBUG] JournalWriteTask - follower: journal writer started 01:21:36,602 PM [follower-writer-64] [DEBUG] JournalWriteTask - follower: waiting to receive commands 01:21:36,602 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/junit-8046194582193304322/follower/TermInfo.properties does not exist 01:21:36,604 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=0, votedFor=null 01:21:36,604 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - follower: Local TermInfo store seeded with TermInfo{term=0} 01:21:36,604 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-8046194582193304322/follower/TermInfo.properties 01:21:36,604 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-8046194582193304322/follower/journal-v1-1.log 01:21:36,604 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-8046194582193304322/follower/metadata-v1-1.log 01:21:36,604 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: no eligible files found 01:21:36,604 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from -1 to -1 01:21:36,604 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - follower: Recovery completed in in 2.484 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,604 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower: Recovery completed and can restore from snapshot 01:21:36,606 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/follower2172] [WARN] [09/04/2025 13:21:36.606] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower2172]: org.opendaylight.controller.cluster.common.actor.Monitor@609ae7ab 01:21:36,607 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower2: Starting recovery with journal batch size 1000 01:21:36,607 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: started executor 01:21:36,607 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: started executor 01:21:36,607 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower2: metadata journal open: replayFrom=1 applyTo=0 01:21:36,607 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower2: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,607 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - follower2: journal open: applyTo=0 01:21:36,608 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower2: recovering to non-persistent 01:21:36,608 PM [follower2-writer-65] [DEBUG] JournalWriteTask - follower2: journal writer started 01:21:36,608 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/junit-8046194582193304322/follower2/TermInfo.properties does not exist 01:21:36,608 PM [follower2-writer-65] [DEBUG] JournalWriteTask - follower2: waiting to receive commands 01:21:36,609 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower2: Set currentTerm=0, votedFor=null 01:21:36,609 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - follower2: Local TermInfo store seeded with TermInfo{term=0} 01:21:36,609 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-8046194582193304322/follower2/TermInfo.properties 01:21:36,610 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-8046194582193304322/follower2/journal-v1-1.log 01:21:36,610 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-8046194582193304322/follower2/metadata-v1-1.log 01:21:36,610 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: no eligible files found 01:21:36,610 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower2: Moving last applied index from -1 to -1 01:21:36,610 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - follower2: Recovery completed in in 2.003 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,610 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower2: Recovery completed and can restore from snapshot 01:21:36,610 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: onRemoveServer: RemoveServer{serverId='follower'}, state: Idle 01:21:36,611 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]] 01:21:36,611 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: Persist data index=0 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,611 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:36,611 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=80e9dd19-03ae-4bef-ace6-e6fe682fe07f}, logIndex: 0, isSendImmediate: true 01:21:36,611 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:36,611 PM [leader-%d386540] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-8046194582193304322/leader/snapshot-0000000068B99260-246F0373.tmp 01:21:36,611 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,611 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower downNode: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,611 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation RemoveServer{serverId='follower'} [WARN] [09/04/2025 13:21:36.611] [test-pekko.actor.default-dispatcher-16] [pekko://test/] unhandled message from TestActor[pekko://test/user/leader173]: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 01:21:36,611 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower2 (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:36,613 PM [leader-%d386540] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-8046194582193304322/leader/snapshot-0000000068B99260-246F0373.v1 01:21:36,613 PM [leader-%d386540] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-8046194582193304322/leader/snapshot-0000000068B99260-246F0373.v1 01:21:36,613 PM [leader-%d386540] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-8046194582193304322/leader/TermInfo.properties 01:21:36,613 PM [leader-%d386540] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-8046194582193304322/leader/journal-v1-1.log 01:21:36,613 PM [leader-%d386540] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-8046194582193304322/leader/metadata-v1-1.log 01:21:36,613 PM [leader-%d386540] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.611255155Z, file=/tmp/junit-8046194582193304322/leader/snapshot-0000000068B99260-246F0373.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,613 PM [leader-%d386540] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:36,614 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:36,614 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:36,614 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower2: Set currentTerm=1, votedFor=null 01:21:36,614 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower2: Last sync leader does not match current leader leader, need to catch up to -1 01:21:36,614 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower2 updated: matchIndex: -1, nextIndex: 0 01:21:36,614 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,614 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower follower2 01:21:36,614 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower follower2: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} [WARN] [09/04/2025 13:21:36.614] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower2172] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower2, initialSyncDone=false] 01:21:36,614 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower2 (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:36,614 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower2 (Follower): Number of entries to be appended = 1 01:21:36,614 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower2 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:21:36,614 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower2 (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,614 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower2: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:36,615 PM [follower2-%d789182] [DEBUG] RaftStorage - follower2: starting snapshot writeout to /tmp/junit-8046194582193304322/follower2/snapshot-0000000068B99260-24A888C4.tmp 01:21:36,615 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - follower2: Updated server config: isVoting: true, peers: [PeerInfo [id=leader, address=pekko://test/user/leader173, votingState=VOTING], PeerInfo [id=downNode, address=, votingState=VOTING]] 01:21:36,615 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower2 (Follower): Log size is now 1 01:21:36,615 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower2 (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=follower2, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:36,615 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower2: Lagging 0 entries behind leader leader and reached -1 (of expected -1) 01:21:36,615 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower2: awaiting more completions to resolve 1 synchronized callback(s) 01:21:36,615 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower2 updated: matchIndex: 0, nextIndex: 1 01:21:36,615 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Applying to log - commitIndex: 0, lastAppliedIndex: -1 01:21:36,615 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 0 01:21:36,615 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to 0 01:21:36,615 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,615 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,615 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} has been successfully replicated to a majority of followers [WARN] [09/04/2025 13:21:36.615] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower171] unhandled message from TestActor[pekko://test/user/leader173]: ServerRemoved [serverId=follower] 01:21:36,616 PM [follower2-%d789182] [DEBUG] RaftStorage - follower2: finished snapshot writeout to /tmp/junit-8046194582193304322/follower2/snapshot-0000000068B99260-24A888C4.v1 01:21:36,616 PM [follower2-%d789182] [DEBUG] RaftStorage - follower2: selected SNAPSHOT_V1 to handle file /tmp/junit-8046194582193304322/follower2/snapshot-0000000068B99260-24A888C4.v1 01:21:36,616 PM [follower2-%d789182] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-8046194582193304322/follower2/TermInfo.properties 01:21:36,616 PM [follower2-%d789182] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-8046194582193304322/follower2/journal-v1-1.log 01:21:36,616 PM [follower2-%d789182] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-8046194582193304322/follower2/metadata-v1-1.log 01:21:36,616 PM [follower2-%d789182] [DEBUG] RaftStorage - follower2: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.615024836Z, file=/tmp/junit-8046194582193304322/follower2/snapshot-0000000068B99260-24A888C4.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,616 PM [follower2-%d789182] [DEBUG] RaftStorageCompleter - follower2: 1 completion(s) pending 01:21:36,616 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower2: running 1 completion(s) 01:21:36,616 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - follower2: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:36,616 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower2: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@33b6da34} 01:21:36,616 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - follower2: synchronized callbacks after 1.751 ms 01:21:36,617 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower2 (Follower): Commit index set to 0 01:21:36,617 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower2: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:36,617 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - follower2 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:21:36,617 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - follower2 (Follower): Setting last applied to 0 01:21:36,617 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - follower2: Moving last applied index from -1 to 0 01:21:36,617 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - follower2: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=true], ServerInfo[peerId=downNode, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} [WARN] [09/04/2025 13:21:36.617] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower2172] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower2, initialSyncDone=true] 01:21:36,662 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServer ending 01:21:36,663 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower170] 01:21:36,674 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader173] 01:21:36,674 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#330,leader-writer-63]/waiting 01:21:36,674 PM [leader-writer-63] [DEBUG] JournalWriteTask - leader: received 1 commands after 80.75 ms 01:21:36,674 PM [leader-writer-63] [DEBUG] JournalWriteTask - leader: completed 1 commands in 51.12 μs 01:21:36,674 PM [leader-writer-63] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:36,675 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:36,675 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal closed 01:21:36,675 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:36,675 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:36,675 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leaderCollector174] 01:21:36,676 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector175#1050122832] 01:21:36,676 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower171] 01:21:36,676 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - follower: terminating thread VirtualThread[#331,follower-writer-64]/waiting 01:21:36,676 PM [follower-writer-64] [DEBUG] JournalWriteTask - follower: received 1 commands after 74.72 ms 01:21:36,676 PM [follower-writer-64] [DEBUG] JournalWriteTask - follower: completed 1 commands in 20.09 μs 01:21:36,676 PM [follower-writer-64] [DEBUG] JournalWriteTask - follower: journal writer stopped 01:21:36,677 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower: journal closed 01:21:36,677 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - follower: journal closed 01:21:36,677 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:36,677 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:36,677 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector176#1307221417] 01:21:36,677 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower2172] 01:21:36,677 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - follower2: terminating thread VirtualThread[#332,follower2-writer-65]/waiting 01:21:36,678 PM [follower2-writer-65] [DEBUG] JournalWriteTask - follower2: received 1 commands after 69.92 ms 01:21:36,678 PM [follower2-writer-65] [DEBUG] JournalWriteTask - follower2: completed 1 commands in 18.98 μs 01:21:36,678 PM [follower2-writer-65] [DEBUG] JournalWriteTask - follower2: journal writer stopped 01:21:36,678 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower2: journal closed 01:21:36,678 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - follower2: journal closed 01:21:36,678 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower2: stopped executor with 0 remaining tasks 01:21:36,678 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower2: stopped executor with 0 remaining tasks 01:21:36,683 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithNoLeader starting 01:21:36,684 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server179] [WARN] [09/04/2025 13:21:36.684] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server179]: org.opendaylight.controller.cluster.common.actor.Monitor@ac0289c 01:21:36,685 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:36,685 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: started executor 01:21:36,685 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: started executor 01:21:36,686 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:36,686 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,686 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:36,686 PM [new-server-writer-66] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:36,686 PM [new-server-writer-66] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:36,686 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:36,686 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-16884337139501824428/new-server/TermInfo.properties does not exist 01:21:36,689 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:36,689 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:36,689 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16884337139501824428/new-server/TermInfo.properties 01:21:36,689 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16884337139501824428/new-server/journal-v1-1.log 01:21:36,689 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-16884337139501824428/new-server/metadata-v1-1.log 01:21:36,689 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:36,689 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:36,689 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - new-server: Recovery completed in in 2.656 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,689 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:36,689 PM [test-pekko.actor.default-dispatcher-13] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:36,690 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] MockRaftActor - Actor created TestActor[pekko://test/user/leader180] [WARN] [09/04/2025 13:21:36.690] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader180]: org.opendaylight.controller.cluster.common.actor.Monitor@54697265 01:21:36,691 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:36,691 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:36,691 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:36,691 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:36,691 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,691 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:36,691 PM [leader-writer-67] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:36,691 PM [leader-writer-67] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:36,692 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:36,692 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-16884337139501824428/leader/TermInfo.properties does not exist 01:21:36,693 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=0, votedFor=null 01:21:36,693 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=0} 01:21:36,694 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16884337139501824428/leader/TermInfo.properties 01:21:36,694 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16884337139501824428/leader/journal-v1-1.log 01:21:36,694 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-16884337139501824428/leader/metadata-v1-1.log 01:21:36,694 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: no eligible files found 01:21:36,694 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:36,694 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Recovery completed in in 2.051 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,694 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:36,802 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server179, votingMember=true], state: Idle 01:21:36,802 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: No leader - returning NO_LEADER reply 01:21:36,803 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithNoLeader ending 01:21:36,803 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower177] 01:21:36,814 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector178#962918185] 01:21:36,815 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server179] 01:21:36,815 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#335,new-server-writer-66]/waiting 01:21:36,815 PM [new-server-writer-66] [DEBUG] JournalWriteTask - new-server: received 1 commands after 128.7 ms 01:21:36,815 PM [new-server-writer-66] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 29.34 μs 01:21:36,815 PM [new-server-writer-66] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:36,816 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:36,816 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:36,816 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:36,816 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:36,816 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader180] 01:21:36,816 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#336,leader-writer-67]/waiting 01:21:36,816 PM [leader-writer-67] [DEBUG] JournalWriteTask - leader: received 1 commands after 124.7 ms 01:21:36,816 PM [leader-writer-67] [DEBUG] JournalWriteTask - leader: completed 1 commands in 18.77 μs 01:21:36,816 PM [leader-writer-67] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:36,817 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:36,817 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal closed 01:21:36,817 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:36,817 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:36,822 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeServersVotingStatus starting 01:21:36,823 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader184] [WARN] [09/04/2025 13:21:36.823] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader184]: org.opendaylight.controller.cluster.common.actor.Monitor@4966b565 01:21:36,824 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:36,824 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:36,824 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:36,824 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:36,825 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:36,825 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,825 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:36,825 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:36,825 PM [leader-writer-68] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:36,825 PM [leader-writer-68] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:36,825 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-1248706854747811007/leader/TermInfo.properties does not exist 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-1248706854747811007/leader/TermInfo.properties 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-1248706854747811007/leader/journal-v1-1.log 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-1248706854747811007/leader/metadata-v1-1.log 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: no eligible files found 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Recovery completed in in 2.523 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower2, follower] 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:21:36,828 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [09/04/2025 13:21:36.828] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower2183] received dead letter from TestActor[pekko://test/user/leader184]: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} [WARN] [09/04/2025 13:21:36.828] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower182] received dead letter from TestActor[pekko://test/user/leader184]: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]} 01:21:36,830 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/follower182] [WARN] [09/04/2025 13:21:36.831] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower182]: org.opendaylight.controller.cluster.common.actor.Monitor@77eb5092 01:21:36,831 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower: Starting recovery with journal batch size 1000 01:21:36,831 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: started executor 01:21:36,831 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: started executor 01:21:36,832 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower: metadata journal open: replayFrom=1 applyTo=0 01:21:36,833 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,833 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - follower: journal open: applyTo=0 01:21:36,833 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower: recovering to non-persistent 01:21:36,833 PM [follower-writer-69] [DEBUG] JournalWriteTask - follower: journal writer started 01:21:36,833 PM [follower-writer-69] [DEBUG] JournalWriteTask - follower: waiting to receive commands 01:21:36,833 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/junit-1248706854747811007/follower/TermInfo.properties does not exist 01:21:36,835 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=0, votedFor=null 01:21:36,835 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - follower: Local TermInfo store seeded with TermInfo{term=0} 01:21:36,835 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-1248706854747811007/follower/TermInfo.properties 01:21:36,835 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-1248706854747811007/follower/journal-v1-1.log 01:21:36,835 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-1248706854747811007/follower/metadata-v1-1.log 01:21:36,835 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower: no eligible files found 01:21:36,835 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from -1 to -1 01:21:36,835 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - follower: Recovery completed in in 2.201 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,835 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower: Recovery completed and can restore from snapshot 01:21:36,836 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/follower2183] [WARN] [09/04/2025 13:21:36.836] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower2183]: org.opendaylight.controller.cluster.common.actor.Monitor@637970fc 01:21:36,837 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - follower2: Starting recovery with journal batch size 1000 01:21:36,837 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: started executor 01:21:36,837 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: started executor 01:21:36,838 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower2: metadata journal open: replayFrom=1 applyTo=0 01:21:36,838 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - follower2: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,838 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - follower2: journal open: applyTo=0 01:21:36,838 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower2: recovering to non-persistent 01:21:36,838 PM [follower2-writer-70] [DEBUG] JournalWriteTask - follower2: journal writer started 01:21:36,838 PM [follower2-writer-70] [DEBUG] JournalWriteTask - follower2: waiting to receive commands 01:21:36,838 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/junit-1248706854747811007/follower2/TermInfo.properties does not exist 01:21:36,840 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - follower2: Set currentTerm=0, votedFor=null 01:21:36,840 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - follower2: Local TermInfo store seeded with TermInfo{term=0} 01:21:36,840 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-1248706854747811007/follower2/TermInfo.properties 01:21:36,840 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-1248706854747811007/follower2/journal-v1-1.log 01:21:36,840 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-1248706854747811007/follower2/metadata-v1-1.log 01:21:36,840 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - follower2: no eligible files found 01:21:36,840 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower2: Moving last applied index from -1 to -1 01:21:36,840 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - follower2: Recovery completed in in 2.181 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,840 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower2: Recovery completed and can restore from snapshot 01:21:36,841 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={follower2=false, follower=false}, serversVisited=[]], state: Idle 01:21:36,841 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:36,841 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - leader: Updated server config: isVoting: true, peers: [PeerInfo [id=follower2, address=pekko://test/user/follower2183, votingState=NON_VOTING], PeerInfo [id=follower, address=pekko://test/user/follower182, votingState=NON_VOTING]] 01:21:36,841 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]] 01:21:36,842 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: Persist data index=0 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,842 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:36,842 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=03e7ae6f-9e43-41e6-810d-ea1e57537d9a}, logIndex: 0, isSendImmediate: true 01:21:36,842 PM [leader-%d769902] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-1248706854747811007/leader/snapshot-0000000068B99260-32327253.tmp 01:21:36,842 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 0 01:21:36,842 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to 0 01:21:36,842 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,842 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,842 PM [test-pekko.actor.default-dispatcher-15] [INFO] Follower - follower2 (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:36,842 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,843 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation ChangeServersVotingStatus [serverVotingStatusMap={follower2=false, follower=false}, serversVisited=[]] 01:21:36,843 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:36,843 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:36,844 PM [leader-%d769902] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-1248706854747811007/leader/snapshot-0000000068B99260-32327253.v1 01:21:36,844 PM [leader-%d769902] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-1248706854747811007/leader/snapshot-0000000068B99260-32327253.v1 01:21:36,844 PM [leader-%d769902] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-1248706854747811007/leader/TermInfo.properties 01:21:36,844 PM [leader-%d769902] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-1248706854747811007/leader/journal-v1-1.log 01:21:36,844 PM [leader-%d769902] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-1248706854747811007/leader/metadata-v1-1.log 01:21:36,844 PM [leader-%d769902] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.842166867Z, file=/tmp/junit-1248706854747811007/leader/snapshot-0000000068B99260-32327253.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,844 PM [leader-%d769902] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:36,844 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:36,844 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:36,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PropertiesTermInfoStore - follower2: Set currentTerm=1, votedFor=null 01:21:36,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower2: Last sync leader does not match current leader leader, need to catch up to -1 01:21:36,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower2 updated: matchIndex: -1, nextIndex: 0 01:21:36,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower follower2 01:21:36,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower follower2: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} [WARN] [09/04/2025 13:21:36.845] [test-pekko.actor.default-dispatcher-15] [pekko://test/user/follower2183] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower2, initialSyncDone=false] 01:21:36,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower2 (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:36,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower2 (Follower): Number of entries to be appended = 1 01:21:36,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower2 (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:21:36,845 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=1, votedFor=null 01:21:36,845 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower2 (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower2: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] PeerInfos - follower2: Updated server config: isVoting: false, peers: [PeerInfo [id=leader, address=pekko://test/user/leader184, votingState=VOTING], PeerInfo [id=follower, address=pekko://test/user/follower182, votingState=NON_VOTING]] 01:21:36,846 PM [follower2-%d811737] [DEBUG] RaftStorage - follower2: starting snapshot writeout to /tmp/junit-1248706854747811007/follower2/snapshot-0000000068B99260-326E03CC.tmp 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower2 (Follower): Log size is now 1 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower2 (Follower): Commit index set to 0 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower2 (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=follower2, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:36,846 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: -1, nextIndex: 0 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] SyncStatusTracker - follower2: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] Follower - follower2 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActorBehavior - follower2 (Follower): Setting last applied to 0 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] AbstractReplicatedLog - follower2: Moving last applied index from -1 to 0 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftActor - follower2: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,846 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,846 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower2: awaiting more completions to resolve 1 synchronized callback(s) 01:21:36,846 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 0 is present for follower follower 01:21:36,846 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower follower: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:36,846 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower2 updated: matchIndex: 0, nextIndex: 1 01:21:36,846 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower: Last sync leader does not match current leader leader, need to catch up to -1 01:21:36,846 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}]} 01:21:36,847 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): Number of entries to be appended = 1 01:21:36,847 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 01:21:36,847 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,847 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower: Append log entry and persist JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:36,847 PM [follower-%d970209] [DEBUG] RaftStorage - follower: starting snapshot writeout to /tmp/junit-1248706854747811007/follower/snapshot-0000000068B99260-327F07E3.tmp 01:21:36,847 PM [follower2-%d811737] [DEBUG] RaftStorage - follower2: finished snapshot writeout to /tmp/junit-1248706854747811007/follower2/snapshot-0000000068B99260-326E03CC.v1 01:21:36,847 PM [follower2-%d811737] [DEBUG] RaftStorage - follower2: selected SNAPSHOT_V1 to handle file /tmp/junit-1248706854747811007/follower2/snapshot-0000000068B99260-326E03CC.v1 01:21:36,847 PM [follower2-%d811737] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-1248706854747811007/follower2/TermInfo.properties 01:21:36,847 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - follower: Updated server config: isVoting: false, peers: [PeerInfo [id=leader, address=pekko://test/user/leader184, votingState=VOTING], PeerInfo [id=follower2, address=pekko://test/user/follower2183, votingState=NON_VOTING]] 01:21:36,847 PM [follower2-%d811737] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-1248706854747811007/follower2/journal-v1-1.log 01:21:36,847 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): Log size is now 1 01:21:36,847 PM [follower2-%d811737] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-1248706854747811007/follower2/metadata-v1-1.log 01:21:36,847 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): Commit index set to 0 01:21:36,847 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=follower, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:36,847 PM [follower2-%d811737] [DEBUG] RaftStorage - follower2: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.846070732Z, file=/tmp/junit-1248706854747811007/follower2/snapshot-0000000068B99260-326E03CC.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,847 PM [follower2-%d811737] [DEBUG] RaftStorageCompleter - follower2: 1 completion(s) pending 01:21:36,847 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower2: running 1 completion(s) 01:21:36,848 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] ReplicatedLogImpl - follower2: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:36,848 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 0, nextIndex: 1 01:21:36,848 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower2: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@613fcc2b} 01:21:36,848 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 1, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:36,848 PM [test-pekko.actor.default-dispatcher-15] [DEBUG] RaftStorageCompleter - follower2: synchronized callbacks after 1.792 ms [WARN] [09/04/2025 13:21:36.848] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower2183] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower2, initialSyncDone=true] 01:21:36,848 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached 0 (of expected -1) 01:21:36,848 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 0, context.getLastApplied(): -1, lastIndex(): 0 01:21:36,848 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower (Follower): Setting last applied to 0 01:21:36,848 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from -1 to 0 01:21:36,848 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower: Applying state for log index 0 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,848 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower: awaiting more completions to resolve 1 synchronized callback(s) 01:21:36,849 PM [follower-%d970209] [DEBUG] RaftStorage - follower: finished snapshot writeout to /tmp/junit-1248706854747811007/follower/snapshot-0000000068B99260-327F07E3.v1 01:21:36,849 PM [follower-%d970209] [DEBUG] RaftStorage - follower: selected SNAPSHOT_V1 to handle file /tmp/junit-1248706854747811007/follower/snapshot-0000000068B99260-327F07E3.v1 01:21:36,849 PM [follower-%d970209] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-1248706854747811007/follower/TermInfo.properties 01:21:36,849 PM [follower-%d970209] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-1248706854747811007/follower/journal-v1-1.log 01:21:36,849 PM [follower-%d970209] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-1248706854747811007/follower/metadata-v1-1.log 01:21:36,849 PM [follower-%d970209] [DEBUG] RaftStorage - follower: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.847185891Z, file=/tmp/junit-1248706854747811007/follower/snapshot-0000000068B99260-327F07E3.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,849 PM [follower-%d970209] [DEBUG] RaftStorageCompleter - follower: 1 completion(s) pending 01:21:36,849 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower: running 1 completion(s) 01:21:36,849 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower: persist complete JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:36,849 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=false], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@3d87965c} 01:21:36,849 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower: synchronized callbacks after 926.8 μs [WARN] [09/04/2025 13:21:36.849] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower182] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower, initialSyncDone=false] [WARN] [09/04/2025 13:21:36.849] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/follower182] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower, initialSyncDone=true] 01:21:36,894 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - leader: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={follower=true}, serversVisited=[]], state: Idle 01:21:36,894 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:36,894 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PeerInfos - leader: Updated server config: isVoting: true, peers: [PeerInfo [id=follower2, address=pekko://test/user/follower2183, votingState=NON_VOTING], PeerInfo [id=follower, address=pekko://test/user/follower182, votingState=VOTING]] 01:21:36,894 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]] 01:21:36,894 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader: Persist data index=1 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,894 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:36,894 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=f309717f-c71b-492e-be34-5c14ab4550d2}, logIndex: 1, isSendImmediate: true 01:21:36,894 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 1 01:21:36,894 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from 0 to 1 01:21:36,894 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,895 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:36,895 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 1 is present for follower follower2 01:21:36,895 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower follower2: AppendEntries{term=1, leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:36,895 PM [leader-%d769903] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-1248706854747811007/leader/snapshot-0000000068B99260-35553FC9.tmp 01:21:36,895 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:36,895 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 1 is present for follower follower 01:21:36,895 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower follower: AppendEntries{term=1, leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:36,895 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation ChangeServersVotingStatus [serverVotingStatusMap={follower=true}, serversVisited=[]] 01:21:36,895 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:36,895 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): Number of entries to be appended = 1 01:21:36,895 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 01:21:36,895 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,895 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower: Append log entry and persist JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:36,895 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:36,895 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:36,895 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): Number of entries to be appended = 1 01:21:36,895 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 01:21:36,895 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,895 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower2: Append log entry and persist JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:36,895 PM [follower-%d970210] [DEBUG] RaftStorage - follower: starting snapshot writeout to /tmp/junit-1248706854747811007/follower/snapshot-0000000068B99260-35641152.tmp 01:21:36,895 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - follower: Updated server config: isVoting: true, peers: [PeerInfo [id=leader, address=pekko://test/user/leader184, votingState=VOTING], PeerInfo [id=follower2, address=pekko://test/user/follower2183, votingState=NON_VOTING]] 01:21:36,895 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): Log size is now 2 01:21:36,895 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): Commit index set to 1 01:21:36,896 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=follower, logLastIndex=1, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:36,896 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached 1 (of expected -1) 01:21:36,896 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 01:21:36,896 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower (Follower): Setting last applied to 1 01:21:36,896 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from 0 to 1 01:21:36,896 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - follower: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,896 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower: awaiting more completions to resolve 1 synchronized callback(s) 01:21:36,897 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 1, nextIndex: 2 01:21:36,897 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - follower2: Updated server config: isVoting: false, peers: [PeerInfo [id=leader, address=pekko://test/user/leader184, votingState=VOTING], PeerInfo [id=follower, address=pekko://test/user/follower182, votingState=VOTING]] 01:21:36,897 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader: fakeSnapshot purging log to 0 for term 1 01:21:36,897 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): Log size is now 2 01:21:36,897 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): Commit index set to 1 01:21:36,897 PM [follower2-%d811738] [DEBUG] RaftStorage - follower2: starting snapshot writeout to /tmp/junit-1248706854747811007/follower2/snapshot-0000000068B99260-35772B88.tmp 01:21:36,897 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=follower2, logLastIndex=1, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:36,897 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower2: Lagging 0 entries behind leader leader and reached 1 (of expected -1) 01:21:36,897 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower2 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 01:21:36,897 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - follower2 (Follower): Setting last applied to 1 01:21:36,897 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 01:21:36,897 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower2: Moving last applied index from 0 to 1 01:21:36,897 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower2: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:36,897 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower2: awaiting more completions to resolve 1 synchronized callback(s) 01:21:36,897 PM [leader-%d769903] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-1248706854747811007/leader/snapshot-0000000068B99260-35553FC9.v1 01:21:36,897 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower2 updated: matchIndex: 1, nextIndex: 2 01:21:36,897 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower2: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 01:21:36,897 PM [leader-%d769903] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-1248706854747811007/leader/snapshot-0000000068B99260-35553FC9.v1 01:21:36,897 PM [leader-%d769903] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-1248706854747811007/leader/snapshot-0000000068B99260-32327253.v1 01:21:36,897 PM [leader-%d769903] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-1248706854747811007/leader/TermInfo.properties 01:21:36,897 PM [leader-%d769903] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-1248706854747811007/leader/journal-v1-1.log 01:21:36,897 PM [leader-%d769903] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-1248706854747811007/leader/metadata-v1-1.log 01:21:36,897 PM [leader-%d769903] [DEBUG] RaftStorage - leader: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.842166867Z, file=/tmp/junit-1248706854747811007/leader/snapshot-0000000068B99260-32327253.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,898 PM [leader-%d769903] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.894779337Z, file=/tmp/junit-1248706854747811007/leader/snapshot-0000000068B99260-35553FC9.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,898 PM [leader-%d769903] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:36,898 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:36,898 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:36,898 PM [follower-%d970210] [DEBUG] RaftStorage - follower: finished snapshot writeout to /tmp/junit-1248706854747811007/follower/snapshot-0000000068B99260-35641152.v1 01:21:36,898 PM [follower-%d970210] [DEBUG] RaftStorage - follower: selected SNAPSHOT_V1 to handle file /tmp/junit-1248706854747811007/follower/snapshot-0000000068B99260-327F07E3.v1 01:21:36,898 PM [follower-%d970210] [DEBUG] RaftStorage - follower: selected SNAPSHOT_V1 to handle file /tmp/junit-1248706854747811007/follower/snapshot-0000000068B99260-35641152.v1 01:21:36,899 PM [follower-%d970210] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-1248706854747811007/follower/TermInfo.properties 01:21:36,899 PM [follower-%d970210] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-1248706854747811007/follower/journal-v1-1.log 01:21:36,899 PM [follower-%d970210] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-1248706854747811007/follower/metadata-v1-1.log 01:21:36,899 PM [follower-%d970210] [DEBUG] RaftStorage - follower: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.847185891Z, file=/tmp/junit-1248706854747811007/follower/snapshot-0000000068B99260-327F07E3.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,899 PM [follower-%d970210] [DEBUG] RaftStorage - follower: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.895750482Z, file=/tmp/junit-1248706854747811007/follower/snapshot-0000000068B99260-35641152.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,899 PM [follower-%d970210] [DEBUG] RaftStorageCompleter - follower: 1 completion(s) pending 01:21:36,899 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower: running 1 completion(s) 01:21:36,900 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - follower: persist complete JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:36,900 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@7774569a} 01:21:36,900 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - follower: synchronized callbacks after 3.920 ms 01:21:36,901 PM [follower2-%d811738] [DEBUG] RaftStorage - follower2: finished snapshot writeout to /tmp/junit-1248706854747811007/follower2/snapshot-0000000068B99260-35772B88.v1 01:21:36,901 PM [follower2-%d811738] [DEBUG] RaftStorage - follower2: selected SNAPSHOT_V1 to handle file /tmp/junit-1248706854747811007/follower2/snapshot-0000000068B99260-326E03CC.v1 01:21:36,901 PM [follower2-%d811738] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-1248706854747811007/follower2/TermInfo.properties 01:21:36,901 PM [follower2-%d811738] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-1248706854747811007/follower2/journal-v1-1.log 01:21:36,901 PM [follower2-%d811738] [DEBUG] RaftStorage - follower2: skipping unrecognized file /tmp/junit-1248706854747811007/follower2/metadata-v1-1.log 01:21:36,901 PM [follower2-%d811738] [DEBUG] RaftStorage - follower2: selected SNAPSHOT_V1 to handle file /tmp/junit-1248706854747811007/follower2/snapshot-0000000068B99260-35772B88.v1 01:21:36,901 PM [follower2-%d811738] [DEBUG] RaftStorage - follower2: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.846070732Z, file=/tmp/junit-1248706854747811007/follower2/snapshot-0000000068B99260-326E03CC.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,902 PM [follower2-%d811738] [DEBUG] RaftStorage - follower2: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.897002376Z, file=/tmp/junit-1248706854747811007/follower2/snapshot-0000000068B99260-35772B88.v1, size=96, entryCompress=NONE, stateCompress=NONE} 01:21:36,902 PM [follower2-%d811738] [DEBUG] RaftStorageCompleter - follower2: 1 completion(s) pending 01:21:36,902 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower2: running 1 completion(s) 01:21:36,902 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower2: persist complete JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:36,902 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower2: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower2, isVoting=false], ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@6f333896} 01:21:36,902 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - follower2: synchronized callbacks after 4.872 ms 01:21:36,946 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeServersVotingStatus ending 01:21:36,947 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower181] 01:21:36,958 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader184] 01:21:36,958 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#337,leader-writer-68]/waiting 01:21:36,959 PM [leader-writer-68] [DEBUG] JournalWriteTask - leader: received 1 commands after 133.3 ms 01:21:36,959 PM [leader-writer-68] [DEBUG] JournalWriteTask - leader: completed 1 commands in 36.92 μs 01:21:36,959 PM [leader-writer-68] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:36,959 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:36,959 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - leader: journal closed 01:21:36,959 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:36,959 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:36,959 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leaderCollector185] 01:21:36,960 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector186#1096059811] 01:21:36,961 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower182] 01:21:36,961 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EnabledRaftStorage - follower: terminating thread VirtualThread[#338,follower-writer-69]/waiting 01:21:36,961 PM [follower-writer-69] [DEBUG] JournalWriteTask - follower: received 1 commands after 128.3 ms 01:21:36,961 PM [follower-writer-69] [DEBUG] JournalWriteTask - follower: completed 1 commands in 20.49 μs 01:21:36,961 PM [follower-writer-69] [DEBUG] JournalWriteTask - follower: journal writer stopped 01:21:36,961 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower: journal closed 01:21:36,961 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - follower: journal closed 01:21:36,961 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:36,961 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:36,962 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector187#-332451542] 01:21:36,962 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower2183] 01:21:36,962 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EnabledRaftStorage - follower2: terminating thread VirtualThread[#339,follower2-writer-70]/waiting 01:21:36,962 PM [follower2-writer-70] [DEBUG] JournalWriteTask - follower2: received 1 commands after 124.2 ms 01:21:36,962 PM [follower2-writer-70] [DEBUG] JournalWriteTask - follower2: completed 1 commands in 17.45 μs 01:21:36,962 PM [follower2-writer-70] [DEBUG] JournalWriteTask - follower2: journal writer stopped 01:21:36,963 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - follower2: journal closed 01:21:36,963 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - follower2: journal closed 01:21:36,963 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower2: stopped executor with 0 remaining tasks 01:21:36,963 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - follower2: stopped executor with 0 remaining tasks 01:21:36,967 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithExistingFollower starting 01:21:36,968 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server190] [WARN] [09/04/2025 13:21:36.969] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server190]: org.opendaylight.controller.cluster.common.actor.Monitor@b1b56 01:21:36,970 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:36,970 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - new-server: started executor 01:21:36,970 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - new-server: started executor 01:21:36,971 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:36,971 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,971 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:36,971 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:36,971 PM [new-server-writer-71] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:36,971 PM [new-server-writer-71] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:36,971 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - /tmp/junit-408525966910239996/new-server/TermInfo.properties does not exist 01:21:36,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:36,974 PM [test-pekko.actor.default-dispatcher-16] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:36,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/TermInfo.properties 01:21:36,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/journal-v1-1.log 01:21:36,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/metadata-v1-1.log 01:21:36,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:36,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:36,974 PM [test-pekko.actor.default-dispatcher-16] [INFO] Recovery - new-server: Recovery completed in in 2.756 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,974 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:36,974 PM [test-pekko.actor.default-dispatcher-16] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:36,975 PM [main] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from -1 to 2 01:21:36,975 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader191] 01:21:36,976 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor=leader 01:21:36,977 PM [test-pekko.actor.default-dispatcher-12] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:36,977 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: started executor [WARN] [09/04/2025 13:21:36.975] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader191]: org.opendaylight.controller.cluster.common.actor.Monitor@37ac8893 01:21:36,977 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: started executor 01:21:36,978 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:36,978 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:36,978 PM [test-pekko.actor.default-dispatcher-12] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:36,978 PM [leader-writer-72] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:36,978 PM [leader-writer-72] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:36,979 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:36,979 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - /tmp/junit-408525966910239996/leader/TermInfo.properties does not exist 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor=leader 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=leader} 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/TermInfo.properties 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/journal-v1-1.log 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/metadata-v1-1.log 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorage - leader: no eligible files found 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [INFO] Recovery - leader: Recovery completed in in 2.294 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower] 01:21:36,981 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: -1 01:21:36,981 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower: Last sync leader does not match current leader leader, need to catch up to -1 01:21:36,982 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 2, nextIndex: 3 01:21:36,982 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader: fakeSnapshot purging log to 1 for term 1 01:21:36,982 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 1 01:21:36,982 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached 2 (of expected -1) 01:21:36,982 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - follower: fakeSnapshot purging log to 1 for term 1 01:21:36,993 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server190, votingMember=true], state: Idle 01:21:36,993 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server190, votingMember=true] 01:21:36,993 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=2, lastAppliedTerm=1, lastIndex=2, lastTerm=1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on new-server 01:21:36,993 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader: last applied journal index prior to capture: 3 01:21:36,993 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader: support() called 01:21:36,994 PM [leader-%d512960] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:36,994 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating capture snapshot for new server new-server 01:21:36,994 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:36,994 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader: Persising snapshot at EntryInfo[index=2, term=1]/EntryInfo[index=2, term=1] 01:21:36,994 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - leader: support() called 01:21:36,995 PM [leader-%d512961] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-408525966910239996/leader/snapshot-0000000068B99260-3B4F5DA2.tmp 01:21:36,995 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader: Removed in-memory snapshotted entries, adjusted snapshotIndex: 1 and term: 1 01:21:36,995 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): sendInstallSnapshot 01:21:36,995 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Snapshot 157 bytes, total chunks to send: 1 01:21:36,995 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Next chunk: total length=157, offset=0, size=157, hashCode=1546036366 01:21:36,995 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): next snapshot chunk size for follower new-server: 157 01:21:36,995 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractLeader - leader (Leader): InstallSnapshot sent to follower Vector(user, new-server190), Chunk: 1/1 01:21:36,995 PM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - new-server (Follower): Term 1 in "InstallSnapshot{term=1, leaderId=leader, lastIncludedIndex=2, lastIncludedTerm=1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1]}" message is greater than follower's term 0 - updating term 01:21:36,997 PM [leader-%d512961] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-408525966910239996/leader/snapshot-0000000068B99260-3B4F5DA2.v1 01:21:36,997 PM [leader-%d512961] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-408525966910239996/leader/snapshot-0000000068B99260-3B4F5DA2.v1 01:21:36,998 PM [leader-%d512961] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/TermInfo.properties 01:21:36,998 PM [leader-%d512961] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/journal-v1-1.log 01:21:36,998 PM [leader-%d512961] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/metadata-v1-1.log 01:21:36,998 PM [leader-%d512961] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.995057058Z, file=/tmp/junit-408525966910239996/leader/snapshot-0000000068B99260-3B4F5DA2.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:36,998 PM [leader-%d512961] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:36,998 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:36,998 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - leader: snapshot is durable as of 2025-09-04T13:21:36.995057058Z 01:21:36,998 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - leader: Snapshot success, discarding journal entries up to 3 01:21:36,998 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorSnapshotMessageSupport - leader: SnapshotComplete received 01:21:36,998 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=1, votedFor=null 01:21:36,998 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - new-server (Follower): handleInstallSnapshot: InstallSnapshot{term=1, leaderId=leader, lastIncludedIndex=2, lastIncludedTerm=1, datasize=157, chunk=1/1, lastChunkHashCode=OptionalInt[-1]} 01:21:36,998 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - new-server: Last sync leader does not match current leader leader, need to catch up to 2 01:21:36,998 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotTracker - new-server (Follower): addChunk: chunkIndex=1, lastChunkIndex=0, collectedChunks.size=0, lastChunkHashCode=-1 01:21:36,998 PM [test-pekko.actor.default-dispatcher-16] [INFO] Follower - new-server (Follower): Snapshot received from leader: leader 01:21:36,999 PM [test-pekko.actor.default-dispatcher-16] [INFO] SnapshotManager - new-server: Applying snapshot on follower: PlainSnapshotSource{io=MemoryStreamSource{size=157}} 01:21:36,999 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - new-server: support() called [WARN] [09/04/2025 13:21:36.999] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/new-server190] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=false] 01:21:36,999 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - new-server: Converted InstallSnapshot from leader: leader to state 01:21:36,999 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - new-server: last applied journal index prior to persisting applied snapshot: 0 01:21:36,999 PM [test-pekko.actor.default-dispatcher-16] [INFO] MockRaftActor - new-server: support() called 01:21:37,0 PM [new-server-%d782360] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-0002E67A.tmp 01:21:37,4 PM [new-server-%d782360] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-0002E67A.v1 01:21:37,4 PM [new-server-%d782360] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-0002E67A.v1 01:21:37,4 PM [new-server-%d782360] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/TermInfo.properties 01:21:37,4 PM [new-server-%d782360] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/journal-v1-1.log 01:21:37,4 PM [new-server-%d782360] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/metadata-v1-1.log 01:21:37,4 PM [new-server-%d782360] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:37.000190074Z, file=/tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-0002E67A.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:37,4 PM [new-server-%d782360] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:37,4 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:37,4 PM [test-pekko.actor.default-dispatcher-12] [INFO] SnapshotManager - new-server: snapshot is durable as of 2025-09-04T13:21:37.000190074Z 01:21:37,4 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - new-server: support() called 01:21:37,5 PM [test-pekko.actor.default-dispatcher-12] [INFO] MockRaftActor - new-server: applySnapshot called 01:21:37,5 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleInstallSnapshot returning: InstallSnapshotReply{term=1, followerId=new-server, chunkIndex=1, success=true} 01:21:37,5 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - new-server: Snapshot success, discarding journal entries up to 0 01:21:37,5 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorSnapshotMessageSupport - new-server: SnapshotComplete received 01:21:37,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): handleInstallSnapshotReply: InstallSnapshotReply{term=1, followerId=new-server, chunkIndex=1, success=true} 01:21:37,5 PM [test-pekko.actor.default-dispatcher-16] [INFO] AbstractLeader - leader (Leader): Snapshot successfully installed on follower new-server (last chunk 1) - matchIndex set to 2, nextIndex set to 3 01:21:37,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - Sent message UnInitializedFollowerSnapshotReply to self 01:21:37,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - leader: onUnInitializedFollowerSnapshotReply: UnInitializedFollowerSnapshotReply [followerId=new-server] 01:21:37,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - leader: New server configuration : [ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]] 01:21:37,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader: Persist data index=3 term=1 command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:37,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader: Append log entry and persist JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:37,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=9d0f3fe2-b080-494c-9007-e420b44a818b}, logIndex: 3, isSendImmediate: true 01:21:37,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:37,5 PM [leader-%d512962] [DEBUG] RaftStorage - leader: starting snapshot writeout to /tmp/junit-408525966910239996/leader/snapshot-0000000068B99261-0058D0A1.tmp 01:21:37,5 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 3, leaderLastIndex: 3, leaderSnapShotIndex: 1 01:21:37,6 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 3 is present for follower follower 01:21:37,6 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower follower: AppendEntries{term=1, leaderId=leader, prevLogIndex=2, prevLogTerm=1, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 3, leaderLastIndex: 3, leaderSnapShotIndex: 1 01:21:37,6 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): sendAppendEntries: 3 is present for follower new-server 01:21:37,6 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Sending AppendEntries to follower new-server: AppendEntries{term=1, leaderId=leader, prevLogIndex=2, prevLogTerm=1, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=2, prevLogTerm=1, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - leader: Returning OK for operation AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server190, votingMember=true] 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): Number of entries to be appended = 1 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 01:21:37,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleAppendEntries: AppendEntries{term=1, leaderId=leader, prevLogIndex=2, prevLogTerm=1, leaderCommit=2, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true}]} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower: Append log entry and persist JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): Number of entries to be appended = 1 01:21:37,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): After cleanup, lastIndex: 2, entries to be added from: 0 01:21:37,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - test: 1 completion(s) pending 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - test: running 1 completion(s) 01:21:37,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - new-server: Append log entry and persist JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] ReplicatedLogImpl - follower: persist complete JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - test: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@6e0b47e4} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - follower: Updated server config: isVoting: true, peers: [PeerInfo [id=leader, address=, votingState=VOTING], PeerInfo [id=new-server, address=null, votingState=VOTING]] 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): Log size is now 2 01:21:37,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - new-server: Updated server config: isVoting: true, peers: [PeerInfo [id=leader, address=null, votingState=VOTING], PeerInfo [id=follower, address=null, votingState=VOTING]] 01:21:37,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): Log size is now 1 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - follower (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=follower, logLastIndex=3, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=0, raftVersion=5, recipientRaftVersion=5} 01:21:37,6 PM [new-server-%d782361] [DEBUG] RaftStorage - new-server: starting snapshot writeout to /tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-006315B6.tmp 01:21:37,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - new-server (Follower): handleAppendEntries returning : AppendEntriesReply{term=1, success=true, followerId=new-server, logLastIndex=3, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=true, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:37,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 2 (of expected 2) 01:21:37,6 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached 2 (of expected -1) 01:21:37,6 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: awaiting more completions to resolve 1 synchronized callback(s) 01:21:37,6 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 3, nextIndex: 4 01:21:37,6 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 4, leaderLastIndex: 3, leaderSnapShotIndex: 1 01:21:37,6 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for new-server updated: matchIndex: 3, nextIndex: 4 01:21:37,7 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: true, followerNextIndex: 4, leaderLastIndex: 3, leaderSnapShotIndex: 1 01:21:37,7 PM [leader-%d512962] [DEBUG] RaftStorage - leader: finished snapshot writeout to /tmp/junit-408525966910239996/leader/snapshot-0000000068B99261-0058D0A1.v1 01:21:37,7 PM [leader-%d512962] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-408525966910239996/leader/snapshot-0000000068B99261-0058D0A1.v1 01:21:37,8 PM [leader-%d512962] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-408525966910239996/leader/snapshot-0000000068B99260-3B4F5DA2.v1 01:21:37,8 PM [leader-%d512962] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/TermInfo.properties 01:21:37,8 PM [leader-%d512962] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/journal-v1-1.log 01:21:37,8 PM [leader-%d512962] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/metadata-v1-1.log 01:21:37,8 PM [leader-%d512962] [DEBUG] RaftStorage - leader: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:36.995057058Z, file=/tmp/junit-408525966910239996/leader/snapshot-0000000068B99260-3B4F5DA2.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:37,8 PM [leader-%d512962] [DEBUG] RaftStorage - leader: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:37.005820577Z, file=/tmp/junit-408525966910239996/leader/snapshot-0000000068B99261-0058D0A1.v1, size=97, entryCompress=NONE, stateCompress=NONE} 01:21:37,8 PM [leader-%d512962] [DEBUG] RaftStorageCompleter - leader: 1 completion(s) pending 01:21:37,8 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - leader: running 1 completion(s) 01:21:37,8 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - leader: persist complete JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=true} 01:21:37,8 PM [new-server-%d782361] [DEBUG] RaftStorage - new-server: finished snapshot writeout to /tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-006315B6.v1 01:21:37,8 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - leader (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2 01:21:37,8 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 3 01:21:37,8 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from 2 to 3 01:21:37,8 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - leader: Applying state for log index 3 data VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:37,8 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SnapshotManager - leader: fakeSnapshot purging log to 2 for term 1 01:21:37,8 PM [new-server-%d782361] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-0002E67A.v1 01:21:37,8 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorVotingConfigSupport - leader: VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:37,8 PM [new-server-%d782361] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-006315B6.v1 01:21:37,9 PM [new-server-%d782361] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/TermInfo.properties 01:21:37,9 PM [new-server-%d782361] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/journal-v1-1.log 01:21:37,9 PM [new-server-%d782361] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/metadata-v1-1.log 01:21:37,9 PM [new-server-%d782361] [DEBUG] RaftStorage - new-server: deleted snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:37.000190074Z, file=/tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-0002E67A.v1, size=64, entryCompress=NONE, stateCompress=NONE} 01:21:37,9 PM [new-server-%d782361] [DEBUG] RaftStorage - new-server: retaining snapshot SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:37.006493622Z, file=/tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-006315B6.v1, size=97, entryCompress=NONE, stateCompress=NONE} 01:21:37,9 PM [new-server-%d782361] [DEBUG] RaftStorageCompleter - new-server: 1 completion(s) pending 01:21:37,9 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: running 1 completion(s) 01:21:37,9 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - new-server: persist complete JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false} 01:21:37,9 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=3, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@7dfc4039} 01:21:37,9 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - new-server: synchronized callbacks after 2.738 ms [WARN] [09/04/2025 13:21:37.009] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/new-server190] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=true] 01:21:37,213 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): Commit index set to 3 01:21:37,213 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached 3 (of expected -1) 01:21:37,213 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - follower (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): 2, lastIndex(): 3 01:21:37,213 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftActorBehavior - follower (Follower): Setting last applied to 3 01:21:37,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - new-server (Follower): Commit index set to 3 01:21:37,213 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from 2 to 3 01:21:37,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] SyncStatusTracker - new-server: Lagging 0 entries behind leader leader and reached 3 (of expected 2) 01:21:37,213 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SnapshotManager - follower: fakeSnapshot purging log to 2 for term 1 01:21:37,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Follower - new-server (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): 2, lastIndex(): 3 01:21:37,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - new-server (Follower): Setting last applied to 3 01:21:37,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from 2 to 3 01:21:37,213 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - new-server: Applying state for log index 3 data VotingConfig{serverInfo=[ServerInfo[peerId=follower, isVoting=true], ServerInfo[peerId=new-server, isVoting=true], ServerInfo[peerId=leader, isVoting=true]]} 01:21:37,246 PM [main] [DEBUG] RaftStorage - leader: selected SNAPSHOT_V1 to handle file /tmp/junit-408525966910239996/leader/snapshot-0000000068B99261-0058D0A1.v1 01:21:37,246 PM [main] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/TermInfo.properties 01:21:37,246 PM [main] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/journal-v1-1.log 01:21:37,246 PM [main] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-408525966910239996/leader/metadata-v1-1.log 01:21:37,246 PM [main] [DEBUG] RaftStorage - leader: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:37.005820577Z, file=/tmp/junit-408525966910239996/leader/snapshot-0000000068B99261-0058D0A1.v1, size=97, entryCompress=NONE, stateCompress=NONE} as the latest file 01:21:37,247 PM [main] [DEBUG] RaftStorage - new-server: selected SNAPSHOT_V1 to handle file /tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-006315B6.v1 01:21:37,247 PM [main] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/TermInfo.properties 01:21:37,247 PM [main] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/journal-v1-1.log 01:21:37,247 PM [main] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-408525966910239996/new-server/metadata-v1-1.log 01:21:37,247 PM [main] [DEBUG] RaftStorage - new-server: picked SnapshotFileV1{lastIncluded=EntryInfo[index=-1, term=-1], timestamp=2025-09-04T13:21:37.006493622Z, file=/tmp/junit-408525966910239996/new-server/snapshot-0000000068B99261-006315B6.v1, size=97, entryCompress=NONE, stateCompress=NONE} as the latest file 01:21:37,247 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithExistingFollower ending 01:21:37,247 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower188] 01:21:37,259 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector189#-974325117] 01:21:37,260 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server190] 01:21:37,260 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#346,new-server-writer-71]/waiting 01:21:37,260 PM [new-server-writer-71] [DEBUG] JournalWriteTask - new-server: received 1 commands after 289.4 ms 01:21:37,261 PM [new-server-writer-71] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 35.08 μs 01:21:37,261 PM [new-server-writer-71] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:37,261 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:37,261 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:37,261 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:37,261 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:37,261 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader191] 01:21:37,262 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#347,leader-writer-72]/waiting 01:21:37,262 PM [leader-writer-72] [DEBUG] JournalWriteTask - leader: received 1 commands after 283.3 ms 01:21:37,262 PM [leader-writer-72] [DEBUG] JournalWriteTask - leader: completed 1 commands in 17.01 μs 01:21:37,262 PM [leader-writer-72] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:37,262 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:37,262 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal closed 01:21:37,262 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:37,262 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:37,263 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leaderCollector192] 01:21:37,267 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerForwardedToLeader starting 01:21:37,269 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server195] [WARN] [09/04/2025 13:21:37.269] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server195]: org.opendaylight.controller.cluster.common.actor.Monitor@688eb23f 01:21:37,269 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:37,269 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: started executor 01:21:37,270 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: started executor 01:21:37,270 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:37,270 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:37,270 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:37,270 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:37,270 PM [new-server-writer-73] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:37,270 PM [new-server-writer-73] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:37,270 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-17852332613251655775/new-server/TermInfo.properties does not exist 01:21:37,274 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:37,274 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:37,274 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-17852332613251655775/new-server/TermInfo.properties 01:21:37,274 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-17852332613251655775/new-server/journal-v1-1.log 01:21:37,274 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-17852332613251655775/new-server/metadata-v1-1.log 01:21:37,274 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:37,274 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:37,274 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - new-server: Recovery completed in in 3.265 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:37,274 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:37,274 PM [test-pekko.actor.default-dispatcher-13] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:37,275 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] MockRaftActor - Actor created TestActor[pekko://test/user/follower197] [WARN] [09/04/2025 13:21:37.275] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower197]: org.opendaylight.controller.cluster.common.actor.Monitor@79fb02a9 01:21:37,276 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - follower: Starting recovery with journal batch size 1000 01:21:37,276 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: started executor 01:21:37,276 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: started executor 01:21:37,276 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower: metadata journal open: replayFrom=1 applyTo=0 01:21:37,276 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:37,276 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - follower: journal open: applyTo=0 01:21:37,276 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower: recovering to non-persistent 01:21:37,276 PM [follower-writer-74] [DEBUG] JournalWriteTask - follower: journal writer started 01:21:37,276 PM [follower-writer-74] [DEBUG] JournalWriteTask - follower: waiting to receive commands 01:21:37,276 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-17852332613251655775/follower/TermInfo.properties does not exist 01:21:37,278 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=0, votedFor=null 01:21:37,278 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - follower: Local TermInfo store seeded with TermInfo{term=0} 01:21:37,278 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-17852332613251655775/follower/TermInfo.properties 01:21:37,278 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-17852332613251655775/follower/journal-v1-1.log 01:21:37,278 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-17852332613251655775/follower/metadata-v1-1.log 01:21:37,278 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: no eligible files found 01:21:37,278 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from -1 to -1 01:21:37,279 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - follower: Recovery completed in in 2.100 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:37,279 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower: Recovery completed and can restore from snapshot 01:21:37,279 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=0, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:37,281 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=1, votedFor=null 01:21:37,281 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower (Follower): The followers log is empty and the senders prevLogIndex is 0 01:21:37,281 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower (Follower): Follower is out-of-sync so sending negative reply: AppendEntriesReply{term=1, success=false, followerId=follower, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:37,281 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower: Last sync leader does not match current leader leader, need to catch up to -1 01:21:37,281 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - follower: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server195, votingMember=true], state: Idle 01:21:37,281 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower: getLeaderAddress leaderId = leader peerAddress = pekko://test/user/leader196 01:21:37,281 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - follower: Not leader - forwarding to leader ActorSelection[Anchor(pekko://test/), Path(/user/leader196)] [WARN] [09/04/2025 13:21:37.282] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower197] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower, initialSyncDone=false] 01:21:37,330 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerForwardedToLeader ending 01:21:37,330 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower193] 01:21:37,332 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector194#-1035030756] 01:21:37,332 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server195] 01:21:37,332 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#353,new-server-writer-73]/waiting 01:21:37,332 PM [new-server-writer-73] [DEBUG] JournalWriteTask - new-server: received 1 commands after 61.77 ms 01:21:37,332 PM [new-server-writer-73] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 31.11 μs 01:21:37,333 PM [new-server-writer-73] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:37,333 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:37,333 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:37,333 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:37,333 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:37,333 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/leader196#1913791413] 01:21:37,333 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower197] 01:21:37,334 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - follower: terminating thread VirtualThread[#354,follower-writer-74]/waiting 01:21:37,334 PM [follower-writer-74] [DEBUG] JournalWriteTask - follower: received 1 commands after 57.38 ms 01:21:37,334 PM [follower-writer-74] [DEBUG] JournalWriteTask - follower: completed 1 commands in 19.97 μs 01:21:37,334 PM [follower-writer-74] [DEBUG] JournalWriteTask - follower: journal writer stopped 01:21:37,334 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower: journal closed 01:21:37,334 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - follower: journal closed 01:21:37,334 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:37,334 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:37,339 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServerForwardToLeader starting 01:21:37,340 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] MockRaftActor - Actor created TestActor[pekko://test/user/follower200] [WARN] [09/04/2025 13:21:37.340] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/follower200]: org.opendaylight.controller.cluster.common.actor.Monitor@21152be1 01:21:37,341 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - follower: Starting recovery with journal batch size 1000 01:21:37,341 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: started executor 01:21:37,341 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: started executor 01:21:37,342 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower: metadata journal open: replayFrom=1 applyTo=0 01:21:37,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:37,343 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - follower: journal open: applyTo=0 01:21:37,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower: recovering to non-persistent 01:21:37,343 PM [follower-writer-75] [DEBUG] JournalWriteTask - follower: journal writer started 01:21:37,343 PM [follower-writer-75] [DEBUG] JournalWriteTask - follower: waiting to receive commands 01:21:37,343 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-5540097305015806982/follower/TermInfo.properties does not exist 01:21:37,345 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=0, votedFor=null 01:21:37,345 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - follower: Local TermInfo store seeded with TermInfo{term=0} 01:21:37,345 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-5540097305015806982/follower/TermInfo.properties 01:21:37,345 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-5540097305015806982/follower/journal-v1-1.log 01:21:37,345 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: skipping unrecognized file /tmp/junit-5540097305015806982/follower/metadata-v1-1.log 01:21:37,345 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: no eligible files found 01:21:37,345 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - follower: Moving last applied index from -1 to -1 01:21:37,346 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - follower: Recovery completed in in 2.559 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:37,346 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower: Recovery completed and can restore from snapshot 01:21:37,346 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=0, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:37,348 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - follower: Set currentTerm=1, votedFor=null 01:21:37,348 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower (Follower): The followers log is empty and the senders prevLogIndex is 0 01:21:37,348 PM [test-pekko.actor.default-dispatcher-13] [INFO] Follower - follower (Follower): Follower is out-of-sync so sending negative reply: AppendEntriesReply{term=1, success=false, followerId=follower, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:37,348 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - follower: Last sync leader does not match current leader leader, need to catch up to -1 01:21:37,348 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - follower: onRemoveServer: RemoveServer{serverId='follower'}, state: Idle 01:21:37,348 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - follower: getLeaderAddress leaderId = leader peerAddress = pekko://test/user/leader199 01:21:37,348 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - follower: Not leader - forwarding to leader ActorSelection[Anchor(pekko://test/), Path(/user/leader199)] [WARN] [09/04/2025 13:21:37.348] [test-pekko.actor.default-dispatcher-13] [pekko://test/user/follower200] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=follower, initialSyncDone=false] 01:21:37,397 PM [main] [INFO] RaftActorVotingConfigSupportTest - testRemoveServerForwardToLeader ending 01:21:37,397 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower198] 01:21:37,398 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader199] 01:21:37,399 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower200] 01:21:37,399 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - follower: terminating thread VirtualThread[#355,follower-writer-75]/waiting 01:21:37,399 PM [follower-writer-75] [DEBUG] JournalWriteTask - follower: received 1 commands after 56.52 ms 01:21:37,400 PM [follower-writer-75] [DEBUG] JournalWriteTask - follower: completed 1 commands in 27.36 μs 01:21:37,400 PM [follower-writer-75] [DEBUG] JournalWriteTask - follower: journal writer stopped 01:21:37,400 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - follower: journal closed 01:21:37,400 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - follower: journal closed 01:21:37,400 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:37,400 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - follower: stopped executor with 0 remaining tasks 01:21:37,404 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithExistingServer starting 01:21:37,405 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader202] [WARN] [09/04/2025 13:21:37.405] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader202]: org.opendaylight.controller.cluster.common.actor.Monitor@3ef74a72 01:21:37,406 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:37,406 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:37,406 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:37,406 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:37,406 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:37,407 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:37,407 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:37,407 PM [leader-writer-76] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:37,407 PM [leader-writer-76] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:37,407 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:37,407 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-11604852235173057016/leader/TermInfo.properties does not exist 01:21:37,409 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:37,409 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:37,409 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-11604852235173057016/leader/TermInfo.properties 01:21:37,409 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-11604852235173057016/leader/journal-v1-1.log 01:21:37,409 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-11604852235173057016/leader/metadata-v1-1.log 01:21:37,409 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: no eligible files found 01:21:37,409 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:37,409 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Recovery completed in in 2.352 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:37,409 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:37,410 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower] 01:21:37,410 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 01:21:37,410 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=follower, newServerAddress=pekko://test/user/follower201, votingMember=true], state: Idle 01:21:37,410 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=follower, newServerAddress=pekko://test/user/follower201, votingMember=true] 01:21:37,411 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Returning ALREADY_EXISTS for operation AddServer [newServerId=follower, newServerAddress=pekko://test/user/follower201, votingMember=true] 01:21:37,411 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithExistingServer ending 01:21:37,411 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower201] 01:21:37,412 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader202] 01:21:37,412 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#356,leader-writer-76]/waiting 01:21:37,412 PM [leader-writer-76] [DEBUG] JournalWriteTask - leader: received 1 commands after 4.959 ms 01:21:37,412 PM [leader-writer-76] [DEBUG] JournalWriteTask - leader: completed 1 commands in 16.55 μs 01:21:37,412 PM [leader-writer-76] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:37,412 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:37,412 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal closed 01:21:37,412 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:37,412 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:37,414 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeToVotingWithNoLeaderAndElectionTimeout starting 01:21:37,416 PM [main] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=1, votedFor=node1 01:21:37,417 PM [main] [DEBUG] EntryJournalV1 - node1: metadata journal open: replayFrom=1 applyTo=0 01:21:37,417 PM [main] [DEBUG] EntryJournalV1 - node1: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:37,421 PM [main] [DEBUG] EntryJournalV1 - node1: journal closed 01:21:37,423 PM [main] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=1, votedFor=node1 01:21:37,423 PM [main] [DEBUG] EntryJournalV1 - node2: metadata journal open: replayFrom=1 applyTo=0 01:21:37,423 PM [main] [DEBUG] EntryJournalV1 - node2: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:37,427 PM [main] [DEBUG] EntryJournalV1 - node2: journal closed 01:21:37,428 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/node1] [WARN] [09/04/2025 13:21:37.428] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/node1]: org.opendaylight.controller.cluster.common.actor.Monitor@237508d9 01:21:37,429 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - node1: Starting recovery with journal batch size 1000 01:21:37,429 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: started executor 01:21:37,429 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: started executor 01:21:37,430 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node1: metadata journal open: replayFrom=1 applyTo=0 01:21:37,431 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node1: journal open: firstIndex=1 lastIndex=1 replayFrom=1 fileEntries=0 01:21:37,431 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - node1: journal open: applyTo=0 01:21:37,431 PM [node1-writer-77] [DEBUG] JournalWriteTask - node1: journal writer started 01:21:37,431 PM [node1-writer-77] [DEBUG] JournalWriteTask - node1: waiting to receive commands 01:21:37,431 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - node1: starting journal recovery 01:21:37,431 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=1, votedFor=node1 01:21:37,431 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-9310414681177115800/node1/TermInfo.properties 01:21:37,431 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-9310414681177115800/node1/journal-v1-1.log 01:21:37,431 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-9310414681177115800/node1/metadata-v1-1.log 01:21:37,431 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: no eligible files found 01:21:37,431 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from -1 to -1 01:21:37,431 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - node1: applying entries up to 0 01:21:37,432 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - node1: recovered journal DefaultLogEntry[index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=true]]}] 01:21:37,432 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - node1: Updated server config: isVoting: false, peers: [PeerInfo [id=node2, address=null, votingState=VOTING]] 01:21:37,432 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from -1 to -1 01:21:37,432 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - node1: Recovery completed in in 1.431 ms: last log index = 0, last log term = 1, napshot index = -1, snapshot term = -1, journal size = 1 01:21:37,432 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - node1: Recovery completed and cannot restore from snapshot 01:21:37,433 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/node2] [WARN] [09/04/2025 13:21:37.434] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/node2]: org.opendaylight.controller.cluster.common.actor.Monitor@23aa854 01:21:37,434 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - node2: Starting recovery with journal batch size 1000 01:21:37,434 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: started executor 01:21:37,434 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: started executor 01:21:37,435 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node2: metadata journal open: replayFrom=1 applyTo=0 01:21:37,436 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node2: journal open: firstIndex=1 lastIndex=1 replayFrom=1 fileEntries=0 01:21:37,436 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - node2: journal open: applyTo=0 01:21:37,436 PM [node2-writer-78] [DEBUG] JournalWriteTask - node2: journal writer started 01:21:37,436 PM [node2-writer-78] [DEBUG] JournalWriteTask - node2: waiting to receive commands 01:21:37,436 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - node2: starting journal recovery 01:21:37,436 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=1, votedFor=node1 01:21:37,436 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-9310414681177115800/node2/TermInfo.properties 01:21:37,436 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-9310414681177115800/node2/journal-v1-1.log 01:21:37,436 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-9310414681177115800/node2/metadata-v1-1.log 01:21:37,436 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: no eligible files found 01:21:37,436 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from -1 to -1 01:21:37,436 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - node2: applying entries up to 0 01:21:37,437 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - node2: recovered journal DefaultLogEntry[index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=true]]}] 01:21:37,437 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - node2: Updated server config: isVoting: true, peers: [PeerInfo [id=node1, address=null, votingState=NON_VOTING]] 01:21:37,437 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from -1 to -1 01:21:37,437 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - node2: Recovery completed in in 929.5 μs: last log index = 0, last log term = 1, napshot index = -1, snapshot term = -1, journal size = 1 01:21:37,437 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - node2: Recovery completed and cannot restore from snapshot 01:21:37,438 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - node1: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={node1=true}, serversVisited=[]], state: Idle 01:21:37,438 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:37,438 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - node1: Sending local ElectionTimeout to start leader election 01:21:37,438 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - node1: Updated server config: isVoting: true, peers: [PeerInfo [id=node2, address=null, votingState=VOTING]] 01:21:37,438 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:37,438 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - node1 (Follower): Received TimeoutNow - switching to Candidate 01:21:37,438 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - node1 (Candidate): Election: Candidate has following voting peers: [node2] 01:21:37,440 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=2, votedFor=node1 01:21:37,440 PM [test-pekko.actor.default-dispatcher-13] [INFO] Candidate - node1 (Candidate): Starting new election term 2 01:21:37,440 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - node1 (Candidate): Sending RequestVote{term=2, candidateId=node1, lastLogIndex=0, lastLogTerm=1} to peer node2 01:21:37,440 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - node1 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:21:37,552 PM [test-pekko.actor.default-dispatcher-11] [WARN] RaftActorVotingConfigSupport - node1: Leader election timed out - cannot apply operation ChangeServersVotingStatus [serverVotingStatusMap={node1=true}, serversVisited=[]] 01:21:37,552 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - node1: Updated server config: isVoting: false, peers: [PeerInfo [id=node2, address=pekko://test/user/node2, votingState=VOTING]] 01:21:37,553 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - node1: tryToForwardOperationToAnotherServer - servers already visited [] 01:21:37,553 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - node1: Returning NO_LEADER for operation ChangeServersVotingStatus [serverVotingStatusMap={node1=true}, serversVisited=[]] 01:21:37,553 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeToVotingWithNoLeaderAndElectionTimeout ending 01:21:37,554 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower203] 01:21:37,555 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector204#843932739] 01:21:37,556 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/node1] 01:21:37,556 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - node1: terminating thread VirtualThread[#357,node1-writer-77]/waiting 01:21:37,556 PM [node1-writer-77] [DEBUG] JournalWriteTask - node1: received 1 commands after 125.2 ms 01:21:37,556 PM [node1-writer-77] [DEBUG] JournalWriteTask - node1: completed 1 commands in 37.49 μs 01:21:37,556 PM [node1-writer-77] [DEBUG] JournalWriteTask - node1: journal writer stopped 01:21:37,556 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node1: journal closed 01:21:37,556 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - node1: journal closed 01:21:37,556 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: stopped executor with 0 remaining tasks 01:21:37,556 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: stopped executor with 0 remaining tasks 01:21:37,557 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector205#1340409311] 01:21:37,557 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/node2] 01:21:37,557 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - node2: terminating thread VirtualThread[#358,node2-writer-78]/waiting 01:21:37,557 PM [node2-writer-78] [DEBUG] JournalWriteTask - node2: received 1 commands after 121.5 ms 01:21:37,557 PM [node2-writer-78] [DEBUG] JournalWriteTask - node2: completed 1 commands in 19.31 μs 01:21:37,557 PM [node2-writer-78] [DEBUG] JournalWriteTask - node2: journal writer stopped 01:21:37,557 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node2: journal closed 01:21:37,557 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - node2: journal closed 01:21:37,558 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: stopped executor with 0 remaining tasks 01:21:37,558 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: stopped executor with 0 remaining tasks 01:21:37,561 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeToVotingWithNoLeader starting 01:21:37,563 PM [main] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=1, votedFor=downNode1 01:21:37,564 PM [main] [DEBUG] EntryJournalV1 - node1: metadata journal open: replayFrom=1 applyTo=0 01:21:37,564 PM [main] [DEBUG] EntryJournalV1 - node1: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:37,568 PM [main] [DEBUG] EntryJournalV1 - node1: updated meta to JournalMeta[replayFrom=1, applyTo=1] 01:21:37,568 PM [main] [DEBUG] EntryJournalV1 - node1: journal closed 01:21:37,570 PM [main] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=1, votedFor=downNode2 01:21:37,570 PM [main] [DEBUG] EntryJournalV1 - node2: metadata journal open: replayFrom=1 applyTo=0 01:21:37,571 PM [main] [DEBUG] EntryJournalV1 - node2: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:37,574 PM [main] [DEBUG] EntryJournalV1 - node2: updated meta to JournalMeta[replayFrom=1, applyTo=1] 01:21:37,574 PM [main] [DEBUG] EntryJournalV1 - node2: journal closed 01:21:37,575 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/node1] [WARN] [09/04/2025 13:21:37.575] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/node1]: org.opendaylight.controller.cluster.common.actor.Monitor@a78d1d1 01:21:37,576 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - node1: Starting recovery with journal batch size 1000 01:21:37,576 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: started executor 01:21:37,576 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: started executor 01:21:37,577 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node1: metadata journal open: replayFrom=1 applyTo=1 01:21:37,577 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node1: journal open: firstIndex=1 lastIndex=1 replayFrom=1 fileEntries=0 01:21:37,577 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - node1: journal open: applyTo=1 01:21:37,577 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: journal writer started 01:21:37,577 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: waiting to receive commands 01:21:37,578 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - node1: starting journal recovery 01:21:37,578 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=1, votedFor=downNode1 01:21:37,578 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-3470902034908000131/node1/TermInfo.properties 01:21:37,578 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-3470902034908000131/node1/journal-v1-1.log 01:21:37,578 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: skipping unrecognized file /tmp/junit-3470902034908000131/node1/metadata-v1-1.log 01:21:37,578 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: no eligible files found 01:21:37,578 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from -1 to -1 01:21:37,578 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - node1: applying entries up to 1 01:21:37,579 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - node1: recovered journal DefaultLogEntry[index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=true], ServerInfo[peerId=downNode2, isVoting=true]]}] 01:21:37,579 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - node1: Updated server config: isVoting: false, peers: [PeerInfo [id=node2, address=null, votingState=NON_VOTING], PeerInfo [id=downNode2, address=null, votingState=VOTING], PeerInfo [id=downNode1, address=null, votingState=VOTING]] 01:21:37,579 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from -1 to 0 01:21:37,579 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - node1: Recovery completed in in 1.309 ms: last log index = 0, last log term = 1, napshot index = -1, snapshot term = -1, journal size = 1 01:21:37,579 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - node1: Recovery completed and cannot restore from snapshot 01:21:37,581 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$CollectingMockRaftActor - Actor created TestActor[pekko://test/user/node2] [WARN] [09/04/2025 13:21:37.581] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/node2]: org.opendaylight.controller.cluster.common.actor.Monitor@60ea098d 01:21:37,582 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - node2: Starting recovery with journal batch size 1000 01:21:37,582 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: started executor 01:21:37,582 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: started executor 01:21:37,583 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node2: metadata journal open: replayFrom=1 applyTo=1 01:21:37,583 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node2: journal open: firstIndex=1 lastIndex=1 replayFrom=1 fileEntries=0 01:21:37,583 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - node2: journal open: applyTo=1 01:21:37,583 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - node2: starting journal recovery 01:21:37,583 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: journal writer started 01:21:37,583 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=1, votedFor=downNode2 01:21:37,583 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: waiting to receive commands 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-3470902034908000131/node2/TermInfo.properties 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-3470902034908000131/node2/journal-v1-1.log 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: skipping unrecognized file /tmp/junit-3470902034908000131/node2/metadata-v1-1.log 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: no eligible files found 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from -1 to -1 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - node2: applying entries up to 1 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] JournalRecovery - node2: recovered journal DefaultLogEntry[index=0, term=1, command=VotingConfig{serverInfo=[ServerInfo[peerId=node1, isVoting=false], ServerInfo[peerId=node2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=true], ServerInfo[peerId=downNode2, isVoting=true]]}] 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - node2: Updated server config: isVoting: false, peers: [PeerInfo [id=downNode2, address=null, votingState=VOTING], PeerInfo [id=downNode1, address=null, votingState=VOTING], PeerInfo [id=node1, address=null, votingState=NON_VOTING]] 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from -1 to 0 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - node2: Recovery completed in in 926.0 μs: last log index = 0, last log term = 1, napshot index = -1, snapshot term = -1, journal size = 1 01:21:37,584 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - node2: Recovery completed and cannot restore from snapshot 01:21:37,585 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - node1: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={downNode2=false, downNode1=false, node2=true, node1=true}, serversVisited=[]], state: Idle 01:21:37,586 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:37,586 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - node1: Sending local ElectionTimeout to start leader election 01:21:37,586 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - node1: Updated server config: isVoting: true, peers: [PeerInfo [id=node2, address=null, votingState=VOTING], PeerInfo [id=downNode2, address=null, votingState=NON_VOTING], PeerInfo [id=downNode1, address=null, votingState=NON_VOTING]] 01:21:37,586 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy 01:21:37,586 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - node1 (Follower): Received TimeoutNow - switching to Candidate 01:21:37,586 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - node1 (Candidate): Election: Candidate has following voting peers: [node2] 01:21:37,588 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=2, votedFor=node1 01:21:37,588 PM [test-pekko.actor.default-dispatcher-13] [INFO] Candidate - node1 (Candidate): Starting new election term 2 01:21:37,588 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - node1 (Follower) :- Switching from behavior Follower to Candidate, election term: 2 01:21:38,103 PM [test-pekko.actor.default-dispatcher-11] [WARN] RaftActorVotingConfigSupport - node1: Leader election timed out - cannot apply operation ChangeServersVotingStatus [serverVotingStatusMap={downNode2=false, downNode1=false, node2=true, node1=true}, serversVisited=[]] 01:21:38,103 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PeerInfos - node1: Updated server config: isVoting: false, peers: [PeerInfo [id=node2, address=null, votingState=NON_VOTING], PeerInfo [id=downNode2, address=null, votingState=VOTING], PeerInfo [id=downNode1, address=null, votingState=VOTING]] 01:21:38,103 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - node1: tryToForwardOperationToAnotherServer - servers already visited [] 01:21:38,103 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorVotingConfigSupport - node1: Returning NO_LEADER for operation ChangeServersVotingStatus [serverVotingStatusMap={downNode2=false, downNode1=false, node2=true, node1=true}, serversVisited=[]] 01:21:38,105 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - node1: Last sync leader does not match current leader downNode1, need to catch up to 0 [WARN] [09/04/2025 13:21:38.105] [test-pekko.actor.default-dispatcher-11] [pekko://test/deadLetters] received dead letter from TestActor[pekko://test/user/node1]: AppendEntriesReply{term=2, success=true, followerId=node1, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=true, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} [WARN] [09/04/2025 13:21:38.105] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/node1] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=node1, initialSyncDone=false] 01:21:38,713 PM [main] [INFO] PeerInfos - node1: Peer address for peer node2 set to pekko://test/user/node2 01:21:38,713 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - node1: onChangeServersVotingStatus: ChangeServersVotingStatus [serverVotingStatusMap={downNode2=false, downNode1=false, node2=true, node1=true}, serversVisited=[]], state: Idle 01:21:38,713 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - Initiating ChangeServersVotingStatusState 01:21:38,713 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - node1: Sending local ElectionTimeout to start leader election 01:21:38,713 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PeerInfos - node1: Updated server config: isVoting: true, peers: [PeerInfo [id=node2, address=pekko://test/user/node2, votingState=VOTING], PeerInfo [id=downNode2, address=null, votingState=NON_VOTING], PeerInfo [id=downNode1, address=null, votingState=NON_VOTING]] 01:21:38,713 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - node1 (Follower): Received TimeoutNow - switching to Candidate 01:21:38,713 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - node1 (Candidate): Election: Candidate has following voting peers: [node2] 01:21:38,716 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - node1: Set currentTerm=3, votedFor=node1 01:21:38,716 PM [test-pekko.actor.default-dispatcher-13] [INFO] Candidate - node1 (Candidate): Starting new election term 3 01:21:38,716 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Candidate - node1 (Candidate): Sending RequestVote{term=3, candidateId=node1, lastLogIndex=0, lastLogTerm=1} to peer node2 01:21:38,716 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActorBehavior - node1 (Follower) :- Switching from behavior Follower to Candidate, election term: 3 01:21:38,716 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node2 (Follower): Found higher term in RequestVote rpc, verifying whether it's safe to update term. 01:21:38,716 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node2 (Follower): Cluster state: [] 01:21:38,716 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node2 (Follower): Candidate in requestVote:RequestVote{term=3, candidateId=node1, lastLogIndex=0, lastLogTerm=1} with higher term appears reachable, updating term. 01:21:38,716 PM [test-pekko.actor.default-dispatcher-11] [INFO] Follower - node2 (Follower): Term 3 in "RequestVote{term=3, candidateId=node1, lastLogIndex=0, lastLogTerm=1}" message is greater than follower's term 1 - updating term 01:21:38,719 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=3, votedFor=null 01:21:38,719 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node2 (Follower): In requestVote: RequestVote{term=3, candidateId=node1, lastLogIndex=0, lastLogTerm=1} - currentTerm: 3, votedFor: null, lastIndex: 0, lastTerm: 1 01:21:38,721 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] PropertiesTermInfoStore - node2: Set currentTerm=3, votedFor=node1 01:21:38,721 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] RaftActorBehavior - node2 (Follower): requestVote returning: RequestVoteReply{term=3, voteGranted=true} 01:21:38,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - node1 (Candidate): RaftRPC message received RequestVoteReply{term=3, voteGranted=true}, my term is 3 01:21:38,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] Candidate - node1 (Candidate): handleRequestVoteReply: RequestVoteReply{term=3, voteGranted=true}, current voteCount: 1 01:21:38,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - node1 (Leader): Election: Leader has following peers: [node2, downNode2, downNode1] 01:21:38,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - node1 (Leader): Checking sendAppendEntries for follower node2: active: false, followerNextIndex: 0, leaderLastIndex: 0, leaderSnapShotIndex: -1 01:21:38,721 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorBehavior - node1 (Candidate) :- Switching from behavior Candidate to Leader, election term: 3 01:21:38,721 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - node2: Last sync leader does not match current leader node1, need to catch up to -1 01:21:38,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - node1: New leader node1 elected 01:21:38,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - node1: New server configuration : [ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]] 01:21:38,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - node1: Persist data index=1 term=3 command=VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]} 01:21:38,721 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - node1: Append log entry and persist JournaledLogEntry{index=1, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]}, pending=true} [WARN] [09/04/2025 13:21:38.722] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/node2] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=node2, initialSyncDone=false] 01:21:38,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - node1 (Leader): Replicate message: identifier: ServerOperationContextIdentifier{value=f570bc32-33c8-45fb-ba23-a5d4d8830b2e}, logIndex: 1, isSendImmediate: true 01:21:38,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - node1 (Leader): Checking sendAppendEntries for follower node2: active: false, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:38,722 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: received 1 commands after 1.144 s 01:21:38,722 PM [test-pekko.actor.default-dispatcher-11] [DEBUG] SyncStatusTracker - node2: Lagging -1 entries behind leader node1 and reached 0 (of expected -1) 01:21:38,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - node1: Returning OK for operation ChangeServersVotingStatus [serverVotingStatusMap={downNode2=false, downNode1=false, node2=true, node1=true}, serversVisited=[]] [WARN] [09/04/2025 13:21:38.722] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/node2] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=node2, initialSyncDone=true] 01:21:38,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - node1 (Leader): handleAppendEntriesReply - FollowerLogInformation for node2 updated: matchIndex: 0, nextIndex: 1 01:21:38,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - node1 (Leader): Checking sendAppendEntries for follower node2: active: true, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:38,722 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - node1 (Leader): sendAppendEntries: 1 is present for follower node2 01:21:38,723 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - node1 (Leader): Sending AppendEntries to follower node2: AppendEntries{term=3, leaderId=node1, prevLogIndex=0, prevLogTerm=1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]}, pending=true}]} 01:21:38,723 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node2 (Follower): handleAppendEntries: AppendEntries{term=3, leaderId=node1, prevLogIndex=0, prevLogTerm=1, leaderCommit=0, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, entries==[JournaledLogEntry{index=1, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]}, pending=true}]} 01:21:38,723 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - node1 (Leader): sendAppendEntries: 1 is present for follower node2 01:21:38,723 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node2 (Follower): Number of entries to be appended = 1 01:21:38,723 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node2 (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 01:21:38,723 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node2 (Follower): Append entry to log VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]} 01:21:38,723 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - node2: Append log entry and persist JournaledLogEntry{index=1, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]}, pending=true} 01:21:38,723 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PeerInfos - node2: Updated server config: isVoting: true, peers: [PeerInfo [id=downNode2, address=null, votingState=NON_VOTING], PeerInfo [id=downNode1, address=null, votingState=NON_VOTING], PeerInfo [id=node1, address=null, votingState=VOTING]] 01:21:38,723 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node2 (Follower): Log size is now 2 01:21:38,723 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] Follower - node2 (Follower): handleAppendEntries returning : AppendEntriesReply{term=3, success=true, followerId=node2, logLastIndex=1, logLastTerm=3, forceInstallSnapshot=false, needsLeaderAddress=true, payloadVersion=5, raftVersion=5, recipientRaftVersion=5} 01:21:38,723 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - node2: Lagging 0 entries behind leader node1 and reached 0 (of expected -1) 01:21:38,723 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - node2: awaiting more completions to resolve 1 synchronized callback(s) 01:21:38,723 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: received 1 commands after 1.140 s 01:21:38,723 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - node1 (Leader): handleAppendEntriesReply - FollowerLogInformation for node2 updated: matchIndex: 1, nextIndex: 2 01:21:38,724 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - node1 (Leader): Checking sendAppendEntries for follower node2: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: -1 01:21:38,724 PM [node1-writer-79] [DEBUG] RaftStorageCompleter - node1: 1 completion(s) pending 01:21:38,724 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: completed 1 commands in 1.775 ms 01:21:38,724 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: waiting to receive commands 01:21:38,724 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - node1: running 1 completion(s) 01:21:38,724 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] ReplicatedLogImpl - node1: persist complete JournaledLogEntry{index=1, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]}, pending=true} 01:21:38,724 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractLeader - node1 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: 0 01:21:38,724 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorBehavior - node1 (Leader): Setting last applied to 1 01:21:38,724 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] AbstractReplicatedLog - node1: Moving last applied index from 0 to 1 01:21:38,724 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActor - node1: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]} 01:21:38,724 PM [test-pekko.actor.default-dispatcher-16] [INFO] RaftActorVotingConfigSupport - node1: VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]} has been successfully replicated to a majority of followers 01:21:38,724 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: received 1 commands after 272.8 μs 01:21:38,724 PM [node1-writer-79] [DEBUG] EntryJournalV1 - node1: updated meta to JournalMeta[replayFrom=1, applyTo=2] 01:21:38,724 PM [node1-writer-79] [DEBUG] RaftStorageCompleter - node1: 1 completion(s) pending 01:21:38,724 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: completed 1 commands in 273.7 μs 01:21:38,724 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorageCompleter - node1: running 1 completion(s) 01:21:38,724 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: waiting to receive commands 01:21:38,725 PM [node2-writer-80] [DEBUG] RaftStorageCompleter - node2: 1 completion(s) pending 01:21:38,725 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: completed 1 commands in 1.373 ms 01:21:38,725 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: waiting to receive commands 01:21:38,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - node2: running 1 completion(s) 01:21:38,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] ReplicatedLogImpl - node2: persist complete JournaledLogEntry{index=1, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]}, pending=false} 01:21:38,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - node2: completed synchronized callback AppendReceivedCallback{entry=JournaledLogEntry{index=1, term=3, command=VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]}, pending=false}, callback=org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda/0x00000007c07da280@1ef589c5} 01:21:38,725 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] RaftStorageCompleter - node2: synchronized callbacks after 1.713 ms 01:21:38,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - node2 (Follower): Commit index set to 1 01:21:38,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SyncStatusTracker - node2: Lagging 0 entries behind leader node1 and reached 1 (of expected -1) 01:21:38,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] Follower - node2 (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 1, context.getLastApplied(): 0, lastIndex(): 1 01:21:38,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorBehavior - node2 (Follower): Setting last applied to 1 01:21:38,833 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - node2: Moving last applied index from 0 to 1 01:21:38,834 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - node2: Applying state for log index 1 data VotingConfig{serverInfo=[ServerInfo[peerId=node2, isVoting=true], ServerInfo[peerId=downNode2, isVoting=false], ServerInfo[peerId=downNode1, isVoting=false], ServerInfo[peerId=node1, isVoting=true]]} 01:21:38,834 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: received 1 commands after 109.0 ms 01:21:38,834 PM [node2-writer-80] [DEBUG] EntryJournalV1 - node2: updated meta to JournalMeta[replayFrom=1, applyTo=2] 01:21:38,834 PM [node2-writer-80] [DEBUG] RaftStorageCompleter - node2: 1 completion(s) pending 01:21:38,834 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: completed 1 commands in 434.9 μs 01:21:38,834 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: waiting to receive commands 01:21:38,834 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorageCompleter - node2: running 1 completion(s) 01:21:38,925 PM [main] [INFO] RaftActorVotingConfigSupportTest - testChangeToVotingWithNoLeader ending 01:21:38,925 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower206] 01:21:38,927 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector207#-857534409] 01:21:38,927 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/node1] 01:21:38,927 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - node1: terminating thread VirtualThread[#359,node1-writer-79]/waiting 01:21:38,928 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: received 1 commands after 203.3 ms 01:21:38,928 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: completed 1 commands in 30.73 μs 01:21:38,928 PM [node1-writer-79] [DEBUG] JournalWriteTask - node1: journal writer stopped 01:21:38,928 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node1: journal closed 01:21:38,928 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - node1: journal closed 01:21:38,928 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: stopped executor with 0 remaining tasks 01:21:38,928 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node1: stopped executor with 0 remaining tasks 01:21:38,928 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/collector208#330901753] 01:21:38,928 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/node2] 01:21:38,929 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EnabledRaftStorage - node2: terminating thread VirtualThread[#360,node2-writer-80]/waiting 01:21:38,929 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: received 1 commands after 94.25 ms 01:21:38,929 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: completed 1 commands in 19.48 μs 01:21:38,929 PM [node2-writer-80] [DEBUG] JournalWriteTask - node2: journal writer stopped 01:21:38,929 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - node2: journal closed 01:21:38,929 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - node2: journal closed 01:21:38,929 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: stopped executor with 0 remaining tasks 01:21:38,929 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - node2: stopped executor with 0 remaining tasks 01:21:38,936 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithPriorSnapshotCompleteTimeout starting 01:21:38,937 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockNewFollowerRaftActor - Actor created TestActor[pekko://test/user/new-server211] [WARN] [09/04/2025 13:21:38.938] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/new-server211]: org.opendaylight.controller.cluster.common.actor.Monitor@43230cb0 01:21:38,938 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - new-server: Starting recovery with journal batch size 1000 01:21:38,938 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: started executor 01:21:38,938 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: started executor 01:21:38,939 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: metadata journal open: replayFrom=1 applyTo=0 01:21:38,939 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - new-server: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:38,939 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - new-server: journal open: applyTo=0 01:21:38,939 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: recovering to non-persistent 01:21:38,939 PM [new-server-writer-81] [DEBUG] JournalWriteTask - new-server: journal writer started 01:21:38,939 PM [new-server-writer-81] [DEBUG] JournalWriteTask - new-server: waiting to receive commands 01:21:38,939 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-10410260308523164609/new-server/TermInfo.properties does not exist 01:21:38,942 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=0, votedFor=null 01:21:38,942 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - new-server: Local TermInfo store seeded with TermInfo{term=0} 01:21:38,942 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-10410260308523164609/new-server/TermInfo.properties 01:21:38,942 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-10410260308523164609/new-server/journal-v1-1.log 01:21:38,942 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: skipping unrecognized file /tmp/junit-10410260308523164609/new-server/metadata-v1-1.log 01:21:38,942 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - new-server: no eligible files found 01:21:38,942 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - new-server: Moving last applied index from -1 to -1 01:21:38,942 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - new-server: Recovery completed in in 2.519 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:38,942 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - new-server: Recovery completed and can restore from snapshot 01:21:38,942 PM [test-pekko.actor.default-dispatcher-13] [INFO] DefaultConfigParamsImpl - Trying to use custom RaftPolicy org.opendaylight.controller.cluster.raft.policy.DisableElectionsRaftPolicy 01:21:38,943 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupportTest$MockLeaderRaftActor - Actor created TestActor[pekko://test/user/leader212] [WARN] [09/04/2025 13:21:38.943] [test-pekko.actor.default-dispatcher-12] [pekko://test/user/termination-monitor] received dead letter from TestActor[pekko://test/user/leader212]: org.opendaylight.controller.cluster.common.actor.Monitor@5597ad23 01:21:38,943 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:38,943 PM [test-pekko.actor.default-dispatcher-13] [INFO] RaftActor - leader: Starting recovery with journal batch size 1000 01:21:38,943 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:38,943 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: started executor 01:21:38,944 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: metadata journal open: replayFrom=1 applyTo=0 01:21:38,944 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] EntryJournalV1 - leader: journal open: firstIndex=1 lastIndex=0 replayFrom=1 fileEntries=0 01:21:38,944 PM [test-pekko.actor.default-dispatcher-13] [INFO] EnabledRaftStorage - leader: journal open: applyTo=0 01:21:38,944 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: recovering to non-persistent 01:21:38,944 PM [leader-writer-82] [DEBUG] JournalWriteTask - leader: journal writer started 01:21:38,944 PM [leader-writer-82] [DEBUG] JournalWriteTask - leader: waiting to receive commands 01:21:38,944 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - /tmp/junit-10410260308523164609/leader/TermInfo.properties does not exist 01:21:38,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] PropertiesTermInfoStore - leader: Set currentTerm=1, votedFor= 01:21:38,947 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Local TermInfo store seeded with TermInfo{term=1, votedFor=} 01:21:38,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-10410260308523164609/leader/TermInfo.properties 01:21:38,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-10410260308523164609/leader/journal-v1-1.log 01:21:38,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: skipping unrecognized file /tmp/junit-10410260308523164609/leader/metadata-v1-1.log 01:21:38,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftStorage - leader: no eligible files found 01:21:38,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Moving last applied index from -1 to -1 01:21:38,947 PM [test-pekko.actor.default-dispatcher-13] [INFO] Recovery - leader: Recovery completed in in 3.019 ms: last log index = -1, last log term = -1, napshot index = -1, snapshot term = -1, journal size = 0 01:21:38,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Recovery completed and can restore from snapshot 01:21:38,947 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [] 01:21:38,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActor - leader: Take a snapshot of current state. lastReplicatedLog is null and replicatedToAllIndex is -1 01:21:38,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractReplicatedLog - leader: Capturing Snapshot : lastLogEntry is null. Using snapshot values lastAppliedIndex -1 and lastAppliedTerm -1 instead. 01:21:38,948 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=true] 01:21:38,948 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: captured snapshot at lastSequenceNumber: 0 01:21:38,948 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Persising snapshot at EntryInfo[index=-1, term=-1]/EntryInfo[index=-1, term=-1] 01:21:38,948 PM [test-pekko.actor.default-dispatcher-13] [INFO] MockRaftActor - leader: support() called 01:21:38,949 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: user triggered or root overwrite snapshot encountered, trimming log up to last applied index -1 01:21:38,949 PM [test-pekko.actor.default-dispatcher-13] [INFO] SnapshotManager - leader: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1 01:21:38,959 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: onAddServer: AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server211, votingMember=true], state: Idle 01:21:38,959 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Initiating AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server211, votingMember=true] 01:21:38,959 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] SnapshotManager - leader: captureToInstall should not be called in state PersistCapture[lastJournalIndex=0, request=CaptureSnapshot [lastAppliedIndex=-1, lastAppliedTerm=-1, lastIndex=-1, lastTerm=-1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=true]] 01:21:38,959 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] RaftActorVotingConfigSupport - leader: Snapshot already in progress - waiting for completion 01:21:39,73 PM [test-pekko.actor.default-dispatcher-12] [INFO] Follower - new-server (Follower): Term 1 in "AppendEntries{term=1, leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=-1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=4, leaderRaftVersion=5, entries==[]}" message is greater than follower's term 0 - updating term 01:21:39,76 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] PropertiesTermInfoStore - new-server: Set currentTerm=1, votedFor=null 01:21:39,76 PM [test-pekko.actor.default-dispatcher-12] [DEBUG] SyncStatusTracker - new-server: Last sync leader does not match current leader leader, need to catch up to -1 01:21:39,76 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for new-server updated: matchIndex: -1, nextIndex: 0 01:21:39,76 PM [test-pekko.actor.default-dispatcher-13] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower new-server: active: false, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 [WARN] [09/04/2025 13:21:39.076] [test-pekko.actor.default-dispatcher-16] [pekko://test/user/new-server211] unhandled message from Actor[pekko://test/deadLetters]: FollowerInitialSyncUpStatus[memberId=new-server, initialSyncDone=false] 01:21:39,172 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - leader: handleInstallSnapshotTimeout for new server new-server 01:21:39,172 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftActorVotingConfigSupport - leader: Returning TIMEOUT for operation AddServer [newServerId=new-server, newServerAddress=pekko://test/user/new-server211, votingMember=true] 01:21:39,173 PM [test-pekko.actor.default-dispatcher-16] [WARN] RaftActorVotingConfigSupport - leader: Timeout occured for new server new-server while waiting for prior snapshot to complete 01:21:39,173 PM [main] [INFO] RaftActorVotingConfigSupportTest - testAddServerWithPriorSnapshotCompleteTimeout ending 01:21:39,173 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower209] 01:21:39,185 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/new-serverCollector210#1947082473] 01:21:39,185 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/new-server211] 01:21:39,185 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EnabledRaftStorage - new-server: terminating thread VirtualThread[#363,new-server-writer-81]/waiting 01:21:39,185 PM [new-server-writer-81] [DEBUG] JournalWriteTask - new-server: received 1 commands after 245.9 ms 01:21:39,185 PM [new-server-writer-81] [DEBUG] JournalWriteTask - new-server: completed 1 commands in 33.26 μs 01:21:39,185 PM [new-server-writer-81] [DEBUG] JournalWriteTask - new-server: journal writer stopped 01:21:39,186 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - new-server: journal closed 01:21:39,186 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - new-server: journal closed 01:21:39,186 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:39,186 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - new-server: stopped executor with 0 remaining tasks 01:21:39,186 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader212] 01:21:39,186 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EnabledRaftStorage - leader: terminating thread VirtualThread[#364,leader-writer-82]/waiting 01:21:39,186 PM [leader-writer-82] [DEBUG] JournalWriteTask - leader: received 1 commands after 241.7 ms 01:21:39,186 PM [leader-writer-82] [DEBUG] JournalWriteTask - leader: completed 1 commands in 9.141 μs 01:21:39,186 PM [leader-writer-82] [DEBUG] JournalWriteTask - leader: journal writer stopped 01:21:39,186 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] EntryJournalV1 - leader: journal closed 01:21:39,186 PM [test-pekko.actor.default-dispatcher-16] [INFO] EnabledRaftStorage - leader: journal closed 01:21:39,186 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:39,186 PM [test-pekko.actor.default-dispatcher-16] [DEBUG] RaftStorage - leader: stopped executor with 0 remaining tasks 01:21:39,188 PM [test-pekko.actor.default-dispatcher-13] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [INFO] [09/04/2025 13:21:39.188] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason] [WARN] [09/04/2025 13:21:39.190] [test-pekko.actor.default-dispatcher-16] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.171.38:17355) [INFO] [09/04/2025 13:21:39.190] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.38:17355] - Exiting completed 01:21:39,191 PM [test-pekko.actor.default-dispatcher-16] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.38:17355] - Exiting completed 01:21:39,191 PM [test-pekko.actor.default-dispatcher-16] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.38:17355] - Shutting down... [INFO] [09/04/2025 13:21:39.191] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.38:17355] - Shutting down... [INFO] [09/04/2025 13:21:39.191] [test-pekko.actor.internal-dispatcher-4] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.38:17355] - Successfully shut down 01:21:39,191 PM [test-pekko.actor.default-dispatcher-16] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.38:17355] - Successfully shut down [INFO] [09/04/2025 13:21:39.193] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.171.38:17355/system/remoting-terminator] Shutting down remote daemon. 01:21:39,193 PM [test-pekko.actor.default-dispatcher-13] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. [INFO] [09/04/2025 13:21:39.193] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.171.38:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. 01:21:39,193 PM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports. [INFO] [09/04/2025 13:21:39.196] [test-pekko.remote.default-remote-dispatcher-7] [pekko://test@10.30.171.38:17355/system/remoting-terminator] Remoting shut down. 01:21:39,196 PM [test-pekko.actor.default-dispatcher-13] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down. [DEBUG] [09/04/2025 13:21:39.196] [test-pekko.actor.internal-dispatcher-10] [EventStream] shutting down: StandardOutLogger