[DEBUG] [07/05/2025 09:24:12.837] [main] [EventStream(pekko://test)] logger log1-TestEventListener started
09:24:12,838 AM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started
[DEBUG] [07/05/2025 09:24:12.838] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started
[DEBUG] [07/05/2025 09:24:12.838] [main] [EventStream(pekko://test)] Default Loggers started
[INFO] [07/05/2025 09:24:12.853] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.247:17355] with UID [-7701205439950975876]
09:24:12,854 AM [test-pekko.actor.default-dispatcher-5] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.247:17355] with UID [-7701205439950975876]
[INFO] [07/05/2025 09:24:12.854] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Starting up, Pekko version [1.0.3] ...
09:24:12,854 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Starting up, Pekko version [1.0.3] ...
[INFO] [07/05/2025 09:24:12.856] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Registered cluster JMX MBean [pekko:type=Cluster]
09:24:12,856 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Registered cluster JMX MBean [pekko:type=Cluster]
[INFO] [07/05/2025 09:24:12.856] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Started up successfully
09:24:12,857 AM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Started up successfully
[INFO] [07/05/2025 09:24:12.857] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing
09:24:12,857 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing
09:24:12,857 AM [test-pekko.actor.default-dispatcher-6] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining
[INFO] [07/05/2025 09:24:12.857] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining
09:24:12,971 AM [main] [INFO] LeaderTest - Starting testHandleMessageForUnknownMessage
09:24:12,971 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:12,971 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:24:12,971 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior579#852963596]
09:24:12,982 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader580]
09:24:12,983 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower581]
09:24:12,986 AM [main] [INFO] LeaderTest - Starting testLaggingFollowerStarvation
09:24:12,987 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:12,987 AM [main] [DEBUG] AbstractLeader - leader585 (Leader): Election: Leader has following peers: [follower586, follower587]
09:24:12,988 AM [main] [DEBUG] AbstractLeader - leader585 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower586 updated: matchIndex: 1, nextIndex: 2
09:24:12,988 AM [main] [DEBUG] AbstractLeader - leader585 (Leader): Applying to log - commitIndex: 1, lastAppliedIndex: -1
09:24:12,988 AM [main] [DEBUG] RaftActorBehavior - leader585 (Leader): Setting last applied to 0
09:24:12,988 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 0
09:24:12,988 AM [main] [DEBUG] RaftActorBehavior - leader585 (Leader): Setting last applied to 1
09:24:12,988 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 0 to 1
09:24:13,189 AM [main] [DEBUG] AbstractLeader - leader585 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower586 updated: matchIndex: 2, nextIndex: 3
09:24:13,189 AM [main] [DEBUG] AbstractLeader - leader585 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1
09:24:13,189 AM [main] [DEBUG] RaftActorBehavior - leader585 (Leader): Setting last applied to 2
09:24:13,189 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 1 to 2
09:24:13,390 AM [main] [DEBUG] AbstractLeader - leader585 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower586 updated: matchIndex: 3, nextIndex: 4
09:24:13,390 AM [main] [DEBUG] AbstractLeader - leader585 (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2
09:24:13,390 AM [main] [DEBUG] RaftActorBehavior - leader585 (Leader): Setting last applied to 3
09:24:13,390 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 2 to 3
09:24:13,590 AM [main] [DEBUG] AbstractLeader - leader585 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower586 updated: matchIndex: 4, nextIndex: 5
09:24:13,790 AM [main] [INFO] AbstractLeader - leader585 (Leader): handleAppendEntriesReply: follower follower586 lastIndex 5 is ahead of our lastIndex 4 (snapshotIndex -1, snapshotTerm -1) - forcing install snaphot
09:24:13,790 AM [main] [INFO] SnapshotManager - leader585: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=4, lastAppliedTerm=1, lastIndex=4, lastTerm=1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower586
09:24:13,790 AM [main] [DEBUG] SnapshotManager - leader585: last applied journal index prior to capture: 4
09:24:13,791 AM [main] [DEBUG] RaftStorageCompleter - test: 1 completion(s) pending
09:24:13,791 AM [main] [DEBUG] RaftStorageCompleter - test: running 1 completion(s)
09:24:13,791 AM [main] [INFO] SnapshotManager - leader585: Persising snapshot at EntryInfo[index=4, term=1]/EntryInfo[index=4, term=1]
09:24:13,791 AM [main] [INFO] SnapshotManager - leader585: Removed in-memory snapshotted entries, adjusted snapshotIndex: -1 and term: -1
09:24:13,791 AM [main] [DEBUG] AbstractLeader - leader585 (Leader): sendInstallSnapshot
09:24:13,992 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior582#-988054476]
09:24:14,3 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader583]
09:24:14,3 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower584]
09:24:14,3 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower586#312058161]
09:24:14,3 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower587#-506754932]
09:24:14,7 AM [main] [INFO] LeaderTest - Starting testHandleInstallSnapshotReplyWithInvalidChunkIndex
09:24:14,7 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:14,9 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:24:15,9 AM [main] [DEBUG] AbstractLeader - leader (Leader): sendInstallSnapshot
09:24:20,25 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior588#213467014]
09:24:20,36 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader589]
09:24:20,37 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower590]
09:24:20,40 AM [main] [INFO] LeaderTest - Starting testLeaderAddressInAppendEntries
09:24:20,40 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:20,40 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to -1
09:24:20,41 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:24:20,41 AM [main] [DEBUG] AbstractLeader - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1
09:24:25,54 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior591#1712004254]
09:24:25,66 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader592]
09:24:25,66 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower593]
09:24:25,73 AM [main] [INFO] LeaderTest - Starting testHandleRequestVoteReply
09:24:25,73 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:25,73 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:24:25,73 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior594#-553433414]
09:24:25,84 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader595]
09:24:25,84 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower596]
09:24:25,86 AM [main] [INFO] LeaderTest - Starting testHandleAppendEntriesReplyFailureWithFollowersLogBehindTheLeader
09:24:25,86 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:25,86 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 2
09:24:25,86 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:25,86 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 0
09:24:25,86 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:24:30,104 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior597#728883262]
09:24:30,115 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader598]
09:24:30,115 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower599]
09:24:30,121 AM [main] [INFO] LeaderTest - Starting testTransferLeadershipWithFollowerInitiallyOutOfSync
09:24:30,122 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:30,122 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:24:35,138 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior600#-1732284134]
09:24:35,149 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader601]
09:24:35,149 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower602]
09:24:35,155 AM [main] [INFO] LeaderTest - Starting testInstallSnapshot
09:24:35,155 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:35,156 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 1 to 3
09:24:35,156 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:24:40,172 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior603#271055627]
09:24:40,183 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader604]
09:24:40,183 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower605]
09:24:40,190 AM [main] [INFO] LeaderTest - Starting testFollowerCatchUpWithAppendEntriesMaxDataSizeExceeded
09:24:40,190 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:40,190 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 3
09:24:40,190 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:40,190 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to -1
09:24:40,190 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:24:45,207 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior606#-1771310898]
09:24:45,218 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader607]
09:24:45,219 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower608]
09:24:45,224 AM [main] [INFO] LeaderTest - Starting testHandleAppendEntriesReplyFailureWithFollowersLogTermDifferent
09:24:45,224 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:45,224 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:45,224 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:45,224 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:24:50,240 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior609#-527034201]
09:24:50,251 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader610]
09:24:50,251 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower611]
09:24:50,256 AM [main] [INFO] LeaderTest - Starting testTransferLeadershipWithEmptyLog
09:24:50,256 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:50,257 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:24:55,274 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior612#2044862476]
09:24:55,285 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader613]
09:24:55,285 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower614]
09:24:55,292 AM [main] [INFO] LeaderTest - Starting testSendAppendEntriesSnapshotScenario
09:24:55,294 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:24:55,294 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:00,308 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior615#-947128524]
09:25:00,319 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader616]
09:25:00,319 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower617]
09:25:00,323 AM [main] [INFO] LeaderTest - Starting testLeaderCreatedWithCommitIndexLessThanLastIndex
09:25:00,324 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:00,324 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:00,324 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:05,343 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior618#-1068690676]
09:25:05,354 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader619]
09:25:05,354 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower620]
09:25:05,460 AM [main] [INFO] LeaderTest - Starting testHandleAppendEntriesReplySuccess
09:25:05,460 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:05,460 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:05,460 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:05,460 AM [main] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 2, nextIndex: 3
09:25:05,460 AM [main] [DEBUG] AbstractLeader - leader (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1
09:25:05,460 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 2
09:25:05,460 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 1 to 2
09:25:05,461 AM [main] [DEBUG] SnapshotManager - leader: fakeSnapshot purging log to 1 for term 1
09:25:05,461 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior621#-690171497]
09:25:05,462 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader622]
09:25:05,462 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower623]
09:25:05,465 AM [main] [INFO] LeaderTest - Starting testIsolatedLeaderCheckNoVotingFollowers
09:25:05,465 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:05,465 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:05,465 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:05,465 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior624#307492108]
09:25:05,466 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader625]
09:25:05,466 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower626]
09:25:05,468 AM [main] [INFO] LeaderTest - Starting testHandleAppendEntriesReplyFailureWithFollowersLogEmpty
09:25:05,468 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:05,469 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:05,469 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:05,469 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:10,485 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior627#-1155121473]
09:25:10,496 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader628]
09:25:10,497 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower629]
09:25:10,500 AM [main] [INFO] LeaderTest - Starting testHandleReplicateMessageCommitIndexIncrementedBeforeConsensus
09:25:10,500 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:10,500 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:15,516 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior630#513072345]
09:25:15,527 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader631]
09:25:15,528 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower632]
09:25:15,533 AM [main] [INFO] LeaderTest - Starting testIsolatedLeaderCheckNoFollowers
09:25:15,533 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:15,533 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:25:15,533 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior633#-949178427]
09:25:15,544 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader634]
09:25:15,544 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower635]
09:25:15,546 AM [main] [INFO] LeaderTest - Starting testHandleAppendEntriesReplyWithNewerTerm
09:25:15,546 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:15,546 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:25:15,546 AM [main] [INFO] AbstractLeader - leader (Leader): Term 20 in "AppendEntriesReply{term=20, success=false, followerId=foo, logLastIndex=1000, logLastTerm=10, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=1, raftVersion=5, recipientRaftVersion=5}" message is greater than leader's term 1 - switching to Follower
09:25:15,546 AM [main] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy
09:25:15,546 AM [main] [INFO] RaftActorBehavior - leader (Leader) :- Switching from behavior Leader to Follower, election term: 20
09:25:15,546 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior636#-74130327]
09:25:15,547 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader637]
09:25:15,547 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower638]
09:25:15,549 AM [main] [INFO] LeaderTest - Starting testTransferLeadershipWithFollowerInSync
09:25:15,549 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:15,549 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:20,564 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior639#655524371]
09:25:20,566 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader640]
09:25:20,566 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower641]
09:25:20,570 AM [main] [INFO] LeaderTest - Starting testInitiateForceInstallSnapshot
09:25:20,570 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:20,570 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 1 to 3
09:25:20,571 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:25,583 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior642#-657177701]
09:25:25,594 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader643]
09:25:25,595 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower644]
09:25:25,601 AM [main] [INFO] LeaderTest - Starting testLeaderCreatedWithCommitIndexLessThanFollowersCommitIndex
09:25:25,601 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:25,601 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:25,602 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:30,618 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior645#749457372]
09:25:30,630 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader646]
09:25:30,630 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower647]
09:25:30,734 AM [main] [INFO] LeaderTest - Starting testTransferLeadershipWithFollowerSyncTimeout
09:25:30,734 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:30,735 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:35,751 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior648#-180434167]
09:25:35,762 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader649]
09:25:35,762 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower650]
09:25:35,768 AM [main] [INFO] LeaderTest - Starting testSendingReplicateImmediatelyAfterHeartbeatDoesReplicate
09:25:35,768 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:35,772 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:40,786 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior651#-2080651952]
09:25:40,797 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader652]
09:25:40,797 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower653]
09:25:40,803 AM [main] [INFO] LeaderTest - Starting testLeaderInstallSnapshotState
09:25:40,803 AM [main] [DEBUG] LeaderInstallSnapshotState - test: Snapshot 106 bytes, total chunks to send: 3
09:25:40,803 AM [main] [DEBUG] LeaderInstallSnapshotState - test: Next chunk: total length=106, offset=0, size=50, hashCode=-65379387
09:25:40,803 AM [main] [DEBUG] LeaderInstallSnapshotState - test: Next chunk: total length=106, offset=50, size=50, hashCode=-2011893223
09:25:40,803 AM [main] [DEBUG] LeaderInstallSnapshotState - test: Next chunk: total length=106, offset=100, size=6, hashCode=-1840245320
09:25:40,803 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior654#1659641809]
09:25:40,814 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader655]
09:25:40,814 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower656]
09:25:40,817 AM [main] [INFO] LeaderTest - Starting testIsolatedLeaderCheckTwoFollowersWhenElectionsAreDisabled
09:25:40,817 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:40,818 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower-2, follower-1]
09:25:40,829 AM [main] [WARN] Leader - leader (Leader): At least 1 followers need to be active, Switching leader from Leader to IsolatedLeader
09:25:40,829 AM [main] [DEBUG] AbstractLeader - leader (IsolatedLeader): Election: Leader has following peers: [follower-2, follower-1]
09:25:40,829 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior657#1882919912]
09:25:40,839 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader658]
09:25:40,840 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower659]
09:25:40,842 AM [main] [INFO] LeaderTest - Starting testDuplicateAppendEntriesWillBeSentOnHeartBeat
09:25:40,842 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:40,843 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
[WARN] [07/05/2025 09:25:40.894] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/leader658] received dead letter from TestActor[pekko://test/user/leader658]: org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader$SendHeartBeat@398956e9
[INFO] [pekkoDeadLetter][07/05/2025 09:25:40.894] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/leader658] Message [org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader$SendHeartBeat] from TestActor[pekko://test/user/leader658] to TestActor[pekko://test/user/leader658] was not delivered. [1] dead letters encountered. If this is not an expected behavior then TestActor[pekko://test/user/leader658] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
09:25:40,894 AM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef -  pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader$SendHeartBeat] from TestActor[pekko://test/user/leader658] to TestActor[pekko://test/user/leader658] was not delivered. [1] dead letters encountered. If this is not an expected behavior then TestActor[pekko://test/user/leader658] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
09:25:45,856 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior660#1431295334]
09:25:45,867 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader661]
09:25:45,867 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower662]
09:25:46,72 AM [main] [INFO] LeaderTest - Starting testHandleAppendEntriesReplyWithNewerTermWhenElectionsAreDisabled
09:25:46,72 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:46,72 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:25:46,72 AM [main] [INFO] AbstractLeader - leader (Leader): Term 20 in "AppendEntriesReply{term=20, success=false, followerId=foo, logLastIndex=1000, logLastTerm=10, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=1, raftVersion=5, recipientRaftVersion=5}" message is greater than leader's term 1 - switching to Follower
09:25:46,72 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior663#1728653360]
09:25:46,84 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader664]
09:25:46,84 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower665]
09:25:46,87 AM [main] [INFO] LeaderTest - Starting testLargePayloadSlicingExpiration
09:25:46,87 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:46,87 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to -1
09:25:46,87 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:51,102 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior666#-244253533]
09:25:51,113 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader667]
09:25:51,114 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower668]
09:25:51,120 AM [main] [INFO] LeaderTest - Starting testHeartbeatsAreAlwaysSentIfTheHeartbeatIntervalHasElapsed
09:25:51,120 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:51,122 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:25:56,135 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior669#861506398]
09:25:56,146 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader670]
09:25:56,146 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower671]
09:25:56,152 AM [main] [INFO] LeaderTest - Starting testInitiateInstallSnapshot
09:25:56,152 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:25:56,152 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 1 to 3
09:25:56,152 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:01,165 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior672#-274911742]
09:26:01,176 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader673]
09:26:01,176 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower674]
09:26:01,182 AM [main] [INFO] LeaderTest - Starting testHandleReplicateMessageSendAppendEntriesToFollower
09:26:01,182 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:01,182 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:06,198 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior675#-328682685]
09:26:06,209 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader676]
09:26:06,209 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower677]
09:26:06,215 AM [main] [INFO] LeaderTest - Starting testForceInstallSnapshot
09:26:06,215 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:06,215 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 1 to 3
09:26:06,215 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:11,232 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior678#932834729]
09:26:11,243 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader679]
09:26:11,243 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower680]
09:26:11,249 AM [main] [INFO] LeaderTest - Starting testSendSnapshotfromInstallSnapshotReply
09:26:11,249 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:11,250 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:11,250 AM [main] [DEBUG] AbstractLeader - leader (Leader): sendInstallSnapshot
09:26:16,265 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior681#1434378482]
09:26:16,267 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader682]
09:26:16,267 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower683]
09:26:16,272 AM [main] [INFO] LeaderTest - Starting testHandleAppendEntriesReplyUnknownFollower
09:26:16,272 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:16,272 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:26:16,272 AM [main] [ERROR] AbstractLeader - leader (Leader): handleAppendEntriesReply - unknown follower unkown-follower
09:26:16,272 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior684#939381519]
09:26:16,283 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader685]
09:26:16,283 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower686]
09:26:16,386 AM [main] [INFO] LeaderTest - Starting testThatLeaderSendsAHeartbeatMessageToAllFollowers
09:26:16,386 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:16,386 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:21,401 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior687#916246062]
09:26:21,412 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader688]
09:26:21,413 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower689]
09:26:21,418 AM [main] [INFO] LeaderTest - Starting testHandleReplicateMessageWithHigherTermThanPreviousEntry
09:26:21,419 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:21,419 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 1 to -1
09:26:21,419 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:26,436 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior690#706843710]
09:26:26,447 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader691]
09:26:26,447 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower692]
09:26:26,456 AM [main] [INFO] LeaderTest - Starting testMultipleReplicateWithReplyShouldResultInAppendEntries
09:26:26,456 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:26,457 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:31,474 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior693#-1736037629]
09:26:31,485 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader694]
09:26:31,485 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower695]
09:26:31,691 AM [main] [INFO] LeaderTest - Starting testReplicationConsensusWithNonVotingFollower
09:26:31,691 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:31,691 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to -1
09:26:31,692 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower, nonvoting-follower]
09:26:36,706 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior696#1391657102]
09:26:36,717 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader697]
09:26:36,718 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower698]
09:26:36,718 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/nonvoting-follower699#-1636546753]
09:26:36,723 AM [main] [INFO] LeaderTest - Starting testHandleInstallSnapshotReplyLastChunk
09:26:36,723 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:36,723 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:41,737 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior700#522130054]
09:26:41,748 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader701]
09:26:41,749 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower702]
09:26:41,752 AM [main] [INFO] LeaderTest - Starting testSendAppendEntriesOnAnInProgressInstallSnapshot
09:26:41,752 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:41,752 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:41,753 AM [main] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Snapshot 79 bytes, total chunks to send: 8
09:26:41,753 AM [main] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Next chunk: total length=79, offset=0, size=10, hashCode=1052414894
09:26:46,867 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior703#1304271949]
09:26:46,878 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader704]
09:26:46,878 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower705]
09:26:46,882 AM [main] [INFO] LeaderTest - Starting testHandleReplicateMessageSendAppendEntriesToFollower
09:26:46,882 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:46,883 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:51,897 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior706#-499105544]
09:26:51,908 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader707]
09:26:51,909 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower708]
09:26:51,914 AM [main] [INFO] LeaderTest - Starting testHandleReplicateMessageWhenThereAreNoFollowers
09:26:51,914 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:51,914 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:26:51,914 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 1 to 0
09:26:51,914 AM [main] [DEBUG] AbstractLeader - leader (Leader): Replicate message: identifier: MockIdentifier{value=state-id}, logIndex: 2, isSendImmediate: true
09:26:51,914 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 1
09:26:51,914 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 0 to 1
09:26:51,914 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): Setting last applied to 2
09:26:51,914 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from 1 to 2
09:26:51,914 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior709#1355601892]
09:26:51,925 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader710]
09:26:51,925 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower711]
09:26:51,928 AM [main] [INFO] LeaderTest - Starting testIsolatedLeaderCheckTwoFollowers
09:26:51,928 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:51,929 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower-2, follower-1]
09:26:51,930 AM [main] [WARN] Leader - leader (Leader): At least 1 followers need to be active, Switching leader from Leader to IsolatedLeader
09:26:51,930 AM [main] [DEBUG] AbstractLeader - leader (IsolatedLeader): Election: Leader has following peers: [follower-2, follower-1]
09:26:51,930 AM [main] [INFO] RaftActorBehavior - leader (Leader) :- Switching from behavior Leader to IsolatedLeader, election term: 1
09:26:51,930 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior712#18978601]
09:26:51,930 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader713]
09:26:51,931 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower714]
09:26:51,933 AM [main] [INFO] LeaderTest - Starting testHandleSnapshotSendsPreviousChunksHashCodeWhenSendingNextChunk
09:26:51,933 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:51,934 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:26:51,935 AM [main] [DEBUG] AbstractLeader - leader (Leader): sendInstallSnapshot
[WARN] [07/05/2025 09:26:51.994] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/leader713] received dead letter from TestActor[pekko://test/user/leader713]: org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader$SendHeartBeat@398956e9
[INFO] [pekkoDeadLetter][07/05/2025 09:26:51.994] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/leader713] Message [org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader$SendHeartBeat] from TestActor[pekko://test/user/leader713] to TestActor[pekko://test/user/leader713] was not delivered. [2] dead letters encountered. If this is not an expected behavior then TestActor[pekko://test/user/leader713] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
09:26:51,995 AM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef -  pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader$SendHeartBeat] from TestActor[pekko://test/user/leader713] to TestActor[pekko://test/user/leader713] was not delivered. [2] dead letters encountered. If this is not an expected behavior then TestActor[pekko://test/user/leader713] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
09:26:56,951 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior715#-1191295098]
09:26:56,961 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader716]
09:26:56,961 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower717]
09:26:56,966 AM [main] [INFO] LeaderTest - Starting testReplicationWithPayloadSizeThatExceedsThreshold
09:26:56,966 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:26:56,966 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to -1
09:26:56,966 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: [follower]
09:27:02,23 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior718#309140465]
09:27:02,34 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader719]
09:27:02,34 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower720]
09:27:02,39 AM [main] [INFO] LeaderTest - Starting testLeaderSchedulesHeartbeatsEvenWhenNoFollowersRespondToInitialAppendEntries
09:27:02,41 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:27:02,41 AM [main] [DEBUG] AbstractLeader - leader724 (Leader): Election: Leader has following peers: [follower726, follower725]
09:27:03,42 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior721#-1566411142]
09:27:03,53 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader722]
09:27:03,53 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower723]
09:27:03,53 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader724]
09:27:03,53 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower725#-1780137699]
09:27:03,53 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower726#-1704247470]
09:27:03,56 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:27:03,56 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:27:03,56 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): In requestVote: RequestVote{term=1, candidateId=test, lastLogIndex=10000, lastLogTerm=999} - currentTerm: 1, votedFor: test, lastIndex: 1, lastTerm: 1
09:27:03,56 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): requestVote returning: RequestVoteReply{term=1, voteGranted=true}
09:27:03,57 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior727#538034117]
09:27:03,67 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader728]
09:27:03,68 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower729]
[WARN] [07/05/2025 09:27:03.134] [test-pekko.actor.default-dispatcher-6] [pekko://test/user/leader724] received dead letter from TestActor[pekko://test/user/leader724]: org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader$SendHeartBeat@398956e9
[INFO] [pekkoDeadLetter][07/05/2025 09:27:03.134] [test-pekko.actor.default-dispatcher-11] [pekko://test/user/leader724] Message [org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader$SendHeartBeat] from TestActor[pekko://test/user/leader724] to TestActor[pekko://test/user/leader724] was not delivered. [3] dead letters encountered. If this is not an expected behavior then TestActor[pekko://test/user/leader724] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
09:27:03,135 AM [test-pekko.actor.default-dispatcher-6] [INFO] TestActorRef -  pekkoDeadLetter Message [org.opendaylight.controller.cluster.raft.behaviors.AbstractLeader$SendHeartBeat] from TestActor[pekko://test/user/leader724] to TestActor[pekko://test/user/leader724] was not delivered. [3] dead letters encountered. If this is not an expected behavior then TestActor[pekko://test/user/leader724] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
09:27:03,270 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:27:03,271 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:27:03,271 AM [main] [DEBUG] AbstractLeader - leader (Leader): handleAppendEntries: AppendEntries{term=2, leaderId=leader-1, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=2, replicatedToAllIndex=-1, payloadVersion=0, recipientRaftVersion=5, leaderRaftVersion=5, entries==[SimpleReplicatedLogEntry [index=0, term=2, payload=zero]]}
09:27:03,271 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior730#1947854209]
09:27:03,282 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader731]
09:27:03,282 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower732]
09:27:03,389 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:27:03,390 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:27:03,390 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): In requestVote: RequestVote{term=1, candidateId=test, lastLogIndex=1999, lastLogTerm=1} - currentTerm: 1, votedFor: test, lastIndex: 2000, lastTerm: 1
09:27:03,390 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): requestVote returning: RequestVoteReply{term=1, voteGranted=false}
09:27:03,390 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior733#228682313]
09:27:03,391 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader734]
09:27:03,391 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower735]
09:27:03,394 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:27:03,394 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:27:03,394 AM [main] [INFO] AbstractLeader - leader (Leader): Term 100 in "AppendEntries{term=100, leaderId=leader-1, prevLogIndex=0, prevLogTerm=0, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=0, recipientRaftVersion=5, leaderRaftVersion=5, entries==[]}" message is greater than leader's term 1 - switching to Follower
09:27:03,394 AM [main] [DEBUG] DefaultConfigParamsImpl - No custom RaftPolicy specified. Using DefaultRaftPolicy
09:27:03,394 AM [main] [INFO] RaftActorBehavior - leader (Leader) :- Switching from behavior Leader to Follower, election term: 100
09:27:03,394 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:27:03,394 AM [main] [INFO] AbstractLeader - leader (Leader): Term 100 in "AppendEntriesReply{term=100, success=false, followerId=follower-1, logLastIndex=100, logLastTerm=100, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=0, raftVersion=5, recipientRaftVersion=5}" message is greater than leader's term 1 - switching to Follower
09:27:03,394 AM [main] [INFO] RaftActorBehavior - leader (Leader) :- Switching from behavior Leader to Follower, election term: 100
09:27:03,394 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:27:03,394 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): Found higher term in RequestVote rpc, verifying whether it's safe to update term.
09:27:03,394 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): Cluster state: []
09:27:03,394 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): Candidate in requestVote:RequestVote{term=100, candidateId=candidate-1, lastLogIndex=10, lastLogTerm=100} with higher term appears reachable, updating term.
09:27:03,394 AM [main] [INFO] AbstractLeader - leader (Leader): Term 100 in "RequestVote{term=100, candidateId=candidate-1, lastLogIndex=10, lastLogTerm=100}" message is greater than leader's term 1 - switching to Follower
09:27:03,395 AM [main] [INFO] RaftActorBehavior - leader (Leader) :- Switching from behavior Leader to Follower, election term: 100
09:27:03,395 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:27:03,395 AM [main] [INFO] AbstractLeader - leader (Leader): Term 100 in "RequestVoteReply{term=100, voteGranted=false}" message is greater than leader's term 1 - switching to Follower
09:27:03,395 AM [main] [INFO] RaftActorBehavior - leader (Leader) :- Switching from behavior Leader to Follower, election term: 100
09:27:03,395 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior736#314249634]
09:27:03,396 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader737]
09:27:03,396 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower738]
09:27:03,398 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:27:03,398 AM [main] [DEBUG] AbstractLeader - test (Leader): Election: Leader has following peers: []
09:27:03,398 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 0
09:27:03,398 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 0
09:27:03,398 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:27:03,398 AM [main] [DEBUG] SnapshotManager - test: fakeSnapshot purging log to 0 for term 1
09:27:03,398 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 2
09:27:03,398 AM [main] [DEBUG] SnapshotManager - test: fakeSnapshot purging log to 1 for term 1
09:27:03,398 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 2
09:27:03,398 AM [main] [DEBUG] SnapshotManager - test: fakeSnapshot purging log to 1 for term 1
09:27:03,398 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior739#-1165353959]
09:27:03,399 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader740]
09:27:03,399 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower741]
09:27:03,401 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:27:03,401 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:27:03,401 AM [main] [INFO] RaftActorBehavior - leader (Leader): Cannot append entries because sender's term 100 is less than 1000
09:27:03,401 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior742#-513613847]
09:27:03,402 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader743]
09:27:03,402 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower744]
09:27:03,404 AM [main] [DEBUG] AbstractReplicatedLog - : Moving last applied index from -1 to 1
09:27:03,404 AM [main] [DEBUG] AbstractLeader - leader (Leader): Election: Leader has following peers: []
09:27:03,404 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): In requestVote: RequestVote{term=999, candidateId=test, lastLogIndex=10000, lastLogTerm=999} - currentTerm: 1000, votedFor: null, lastIndex: 1, lastTerm: 1
09:27:03,404 AM [main] [DEBUG] RaftActorBehavior - leader (Leader): requestVote returning: RequestVoteReply{term=1000, voteGranted=false}
09:27:03,404 AM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior745#1504395300]
09:27:03,404 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader746]
09:27:03,404 AM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower747]
[INFO] [07/05/2025 09:27:03.405] [main] [CoordinatedShutdown(pekko://test)] Running CoordinatedShutdown with reason [ActorSystemTerminateReason]
09:27:03,405 AM [test-pekko.actor.default-dispatcher-11] [INFO] CoordinatedShutdown - Running CoordinatedShutdown with reason [ActorSystemTerminateReason]
[WARN] [07/05/2025 09:27:03.405] [test-pekko.actor.default-dispatcher-11] [pekko://test/system/cluster/core/daemon] unhandled message from Actor[pekko://test/deadLetters]: Leave(pekko://test@10.30.171.247:17355)
09:27:03,405 AM [test-pekko.actor.default-dispatcher-6] [INFO] LocalActorRef -  pekkoDeadLetter Message [org.apache.pekko.cluster.ClusterUserAction$Leave] to Actor[pekko://test/system/cluster/core/daemon#2084509939] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
[INFO] [pekkoDeadLetter][07/05/2025 09:27:03.405] [test-pekko.actor.default-dispatcher-6] [pekko://test/system/cluster/core/daemon] Message [org.apache.pekko.cluster.ClusterUserAction$Leave] to Actor[pekko://test/system/cluster/core/daemon#2084509939] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'pekko.log-dead-letters' and 'pekko.log-dead-letters-during-shutdown'.
[INFO] [07/05/2025 09:27:03.406] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Exiting completed
[INFO] [07/05/2025 09:27:03.406] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Shutting down...
[INFO] [07/05/2025 09:27:03.406] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.247:17355] - Successfully shut down
09:27:03,406 AM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Exiting completed
09:27:03,406 AM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Shutting down...
09:27:03,406 AM [test-pekko.actor.default-dispatcher-11] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.247:17355] - Successfully shut down
[INFO] [07/05/2025 09:27:03.407] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.171.247:17355/system/remoting-terminator] Shutting down remote daemon.
09:27:03,407 AM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon.
[INFO] [07/05/2025 09:27:03.407] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.171.247:17355/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports.
09:27:03,407 AM [test-pekko.actor.default-dispatcher-11] [INFO] RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports.
[INFO] [07/05/2025 09:27:03.408] [test-pekko.remote.default-remote-dispatcher-8] [pekko://test@10.30.171.247:17355/system/remoting-terminator] Remoting shut down.
09:27:03,409 AM [test-pekko.actor.default-dispatcher-5] [INFO] RemoteActorRefProvider$RemotingTerminator - Remoting shut down.
[DEBUG] [07/05/2025 09:27:03.409] [test-pekko.actor.internal-dispatcher-9] [EventStream] shutting down: StandardOutLogger