[DEBUG] [10/27/2024 15:06:14.786] [main] [EventStream(pekko://test)] logger log1-TestEventListener started 03:06:14,786 PM [test-pekko.actor.default-dispatcher-5] [INFO] Slf4jLogger - Slf4jLogger started [DEBUG] [10/27/2024 15:06:14.786] [main] [EventStream(pekko://test)] logger log2-Slf4jLogger started [DEBUG] [10/27/2024 15:06:14.786] [main] [EventStream(pekko://test)] Default Loggers started [INFO] [10/27/2024 15:06:14.798] [main] [ArteryTransport(pekko://test)] Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.89:17355] with UID [6544854307693795661] 03:06:14,799 PM [test-pekko.actor.default-dispatcher-6] [INFO] ArteryTransport - Remoting started with transport [Artery tcp]; listening on address [pekko://test@10.30.171.89:17355] with UID [6544854307693795661] 03:06:14,799 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.89:17355] - Starting up, Pekko version [1.0.3] ... 03:06:14,801 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.89:17355] - Registered cluster JMX MBean [pekko:type=Cluster] 03:06:14,801 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.89:17355] - Started up successfully 03:06:14,803 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.89:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing 03:06:14,803 PM [test-pekko.actor.default-dispatcher-5] [INFO] Cluster - Cluster Node [pekko://test@10.30.171.89:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining [INFO] [10/27/2024 15:06:14.799] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.89:17355] - Starting up, Pekko version [1.0.3] ... [INFO] [10/27/2024 15:06:14.801] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.89:17355] - Registered cluster JMX MBean [pekko:type=Cluster] [INFO] [10/27/2024 15:06:14.801] [main] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.89:17355] - Started up successfully [INFO] [10/27/2024 15:06:14.803] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.89:17355] - No downing-provider-class configured, manual cluster downing required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#downing [INFO] [10/27/2024 15:06:14.803] [test-pekko.actor.internal-dispatcher-3] [Cluster(pekko://test)] Cluster Node [pekko://test@10.30.171.89:17355] - No seed-nodes configured, manual cluster join required, see https://pekko.apache.org/docs/pekko/current/typed/cluster.html#joining 03:06:14,814 PM [main] [INFO] LeaderTest - Starting testHandleMessageForUnknownMessage 03:06:14,814 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from -1 to 1 03:06:14,814 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Election: Leader has following peers: [] 03:06:14,814 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior251#-215621356] 03:06:14,825 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader252] 03:06:14,825 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower253] 03:06:14,830 PM [main] [INFO] LeaderTest - Starting testLaggingFollowerStarvation 03:06:14,831 PM [main] [DEBUG] MockRaftActorContext - leader257: Moving last applied index from -1 to 1 03:06:14,832 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Election: Leader has following peers: [follower259, follower258] 03:06:14,832 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Checking sendAppendEntries for follower follower259: active: false, followerNextIndex: 1, leaderLastIndex: 4, leaderSnapShotIndex: -1 03:06:14,835 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Checking sendAppendEntries for follower follower258: active: false, followerNextIndex: 1, leaderLastIndex: 4, leaderSnapShotIndex: -1 03:06:14,835 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower258 updated: matchIndex: 1, nextIndex: 2 03:06:14,835 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Checking sendAppendEntries for follower follower258: active: true, followerNextIndex: 2, leaderLastIndex: 4, leaderSnapShotIndex: -1 03:06:14,835 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): sendAppendEntries: 2 is present for follower follower258 03:06:14,835 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Sending AppendEntries to follower follower258: AppendEntries [leaderId=leader257, prevLogIndex=2, prevLogTerm=1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=0, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=3, term=1, payload=3], SimpleReplicatedLogEntry [index=4, term=1, payload=4]]] 03:06:15,36 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower258 updated: matchIndex: 2, nextIndex: 3 03:06:15,36 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Applying to log - commitIndex: 2, lastAppliedIndex: 1 03:06:15,36 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Setting last applied to 2 03:06:15,36 PM [main] [DEBUG] MockRaftActorContext - leader257: Moving last applied index from 1 to 2 03:06:15,36 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Checking sendAppendEntries for follower follower258: active: true, followerNextIndex: 3, leaderLastIndex: 4, leaderSnapShotIndex: -1 03:06:15,36 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): sendAppendEntries: 3 is present for follower follower258 03:06:15,36 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Sending AppendEntries to follower follower258: AppendEntries [leaderId=leader257, prevLogIndex=3, prevLogTerm=1, leaderCommit=2, replicatedToAllIndex=-1, payloadVersion=0, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=4, term=1, payload=4]]] 03:06:15,237 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower258 updated: matchIndex: 3, nextIndex: 4 03:06:15,237 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Applying to log - commitIndex: 3, lastAppliedIndex: 2 03:06:15,237 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Setting last applied to 3 03:06:15,237 PM [main] [DEBUG] MockRaftActorContext - leader257: Moving last applied index from 2 to 3 03:06:15,237 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Checking sendAppendEntries for follower follower258: active: true, followerNextIndex: 4, leaderLastIndex: 4, leaderSnapShotIndex: -1 03:06:15,237 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): sendAppendEntries: 4 is present for follower follower258 03:06:15,437 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): handleAppendEntriesReply - FollowerLogInformation for follower258 updated: matchIndex: 4, nextIndex: 5 03:06:15,438 PM [main] [DEBUG] MockRaftActorContext - leader257 (Leader): Checking sendAppendEntries for follower follower258: active: true, followerNextIndex: 5, leaderLastIndex: 4, leaderSnapShotIndex: -1 03:06:15,638 PM [main] [INFO] MockRaftActorContext - leader257 (Leader): handleAppendEntriesReply: follower follower258 lastIndex 5 is ahead of our lastIndex 4 (snapshotIndex -1, snapshotTerm -1) - forcing install snaphot 03:06:15,638 PM [main] [INFO] MockRaftActorContext - leader257: Initiating snapshot capture CaptureSnapshot [lastAppliedIndex=4, lastAppliedTerm=1, lastIndex=4, lastTerm=1, installSnapshotInitiated=, replicatedToAllIndex=-1, replicatedToAllTerm=-1, unAppliedEntries size=0, mandatoryTrim=false] to install on follower258 03:06:15,638 PM [main] [DEBUG] MockRaftActorContext - leader257: lastSequenceNumber prior to capture: -1 03:06:15,839 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior254#-80986014] 03:06:15,841 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader255] 03:06:15,841 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower256] 03:06:15,841 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower258#-752867453] 03:06:15,842 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/follower259#-2044578388] 03:06:15,845 PM [main] [INFO] LeaderTest - Starting testHandleInstallSnapshotReplyWithInvalidChunkIndex 03:06:15,845 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from -1 to 1 03:06:15,848 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Election: Leader has following peers: [follower] 03:06:15,848 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: 3, leaderLastIndex: 1, leaderSnapShotIndex: -1 03:06:16,850 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): sendInstallSnapshot 03:06:16,850 PM [main] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Snapshot 106 bytes, total chunks to send: 3 03:06:16,850 PM [main] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Next chunk: total length=106, offset=0, size=50, hashCode=-65379387 03:06:16,850 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): next snapshot chunk size for follower follower: 50 03:06:16,851 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): InstallSnapshot sent to follower Vector(user, follower262), Chunk: 1/3 03:06:16,851 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): handleInstallSnapshotReply: InstallSnapshotReply [term=2, followerId=follower, chunkIndex=-1, success=false] 03:06:16,851 PM [main] [ERROR] MockRaftActorContext - leader (Leader): Chunk index -1 in InstallSnapshotReply from follower follower does not match expected index 1 03:06:16,951 PM [main] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Next chunk: total length=106, offset=0, size=50, hashCode=-65379387 03:06:16,951 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): next snapshot chunk size for follower follower: 50 03:06:16,951 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): InstallSnapshot sent to follower Vector(user, follower262), Chunk: 1/3 03:06:16,952 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior260#1910019617] 03:06:16,963 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader261] 03:06:16,963 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower262] 03:06:16,966 PM [main] [INFO] LeaderTest - Starting testLeaderAddressInAppendEntries 03:06:16,966 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from -1 to 1 03:06:16,966 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from 1 to -1 03:06:16,967 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Election: Leader has following peers: [follower] 03:06:16,967 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: -1, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:06:16,967 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: -1, nextIndex: 0 03:06:16,967 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 0, leaderLastIndex: -1, leaderSnapShotIndex: -1 03:06:17,68 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior263#-881993521] 03:06:17,79 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader264] 03:06:17,80 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower265] 03:06:17,83 PM [main] [INFO] LeaderTest - Starting testHandleRequestVoteReply 03:06:17,83 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from -1 to 1 03:06:17,83 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Election: Leader has following peers: [] 03:06:17,83 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior266#395989616] 03:06:17,94 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader267] 03:06:17,94 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower268] 03:06:17,96 PM [main] [INFO] LeaderTest - Starting testHandleAppendEntriesReplyFailureWithFollowersLogBehindTheLeader 03:06:17,97 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from -1 to 1 03:06:17,97 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from 1 to 2 03:06:17,97 PM [main] [DEBUG] MockRaftActorContext - follower: Moving last applied index from -1 to 1 03:06:17,97 PM [main] [DEBUG] MockRaftActorContext - follower: Moving last applied index from 1 to 0 03:06:17,97 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Election: Leader has following peers: [follower] 03:06:17,97 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: 2, leaderLastIndex: 2, leaderSnapShotIndex: -1 03:06:17,97 PM [main] [INFO] MockRaftActorContext - follower (Follower): The log is not empty but the prevLogIndex 1 was not found in it - lastIndex: 0, snapshotIndex: -1, snapshotTerm: -1 03:06:17,97 PM [main] [INFO] MockRaftActorContext - follower (Follower): Follower is out-of-sync so sending negative reply: AppendEntriesReply [term=1, success=false, followerId=follower, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:06:17,97 PM [main] [DEBUG] SyncStatusTracker - follower: Last sync leader does not match current leader leader, need to catch up to -1 03:06:17,97 PM [main] [INFO] MockRaftActorContext - leader (Leader): handleAppendEntriesReply - received unsuccessful reply: AppendEntriesReply [term=1, success=false, followerId=follower, logLastIndex=0, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5], leader snapshotIndex: -1, snapshotTerm: -1, replicatedToAllIndex: -1 03:06:17,97 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 0, nextIndex: 1 03:06:17,97 PM [main] [INFO] MockRaftActorContext - leader (Leader): follower follower appears to be behind the leader from the last snapshot - updated: matchIndex: 0, nextIndex: 1 03:06:17,97 PM [main] [DEBUG] MockRaftActorContext - leader: fakeSnapshot purging log to 0 for term 1 03:06:17,97 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 1, leaderLastIndex: 2, leaderSnapShotIndex: 0 03:06:17,97 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): sendAppendEntries: 1 is present for follower follower 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Sending AppendEntries to follower follower: AppendEntries [leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=2, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=1, term=1, payload=1], SimpleReplicatedLogEntry [index=2, term=1, payload=2]]] 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): handleAppendEntries: AppendEntries [leaderId=leader, prevLogIndex=0, prevLogTerm=1, leaderCommit=2, replicatedToAllIndex=0, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=1, term=1, payload=1], SimpleReplicatedLogEntry [index=2, term=1, payload=2]]] 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Number of entries to be appended = 2 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): After cleanup, lastIndex: 0, entries to be added from: 0 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Append entry to log 1 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Append entry to log 2 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Log size is now 3 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Commit index set to 2 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): handleAppendEntries returning : AppendEntriesReply [term=1, success=true, followerId=follower, logLastIndex=2, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 2, nextIndex: 3 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - leader: fakeSnapshot purging log to 1 for term 1 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 1 03:06:17,98 PM [main] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached 2 (of expected -1) 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 2, context.getLastApplied(): 0, lastIndex(): 2 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Setting last applied to 1 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower: Moving last applied index from 0 to 1 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Setting last applied to 2 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower: Moving last applied index from 1 to 2 03:06:17,98 PM [main] [DEBUG] MockRaftActorContext - follower: fakeSnapshot purging log to 0 for term 1 03:06:17,98 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior269#232258250] 03:06:17,109 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader270] 03:06:17,109 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower271] 03:06:17,111 PM [main] [INFO] LeaderTest - Starting testTransferLeadershipWithFollowerInitiallyOutOfSync 03:06:17,111 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from -1 to 1 03:06:17,111 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Election: Leader has following peers: [follower] 03:06:17,111 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 03:06:17,182 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Attempting to transfer leadership 03:06:17,183 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: 1, leaderLastIndex: 1, leaderSnapShotIndex: -1 03:06:17,184 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: -1, nextIndex: 0 03:06:17,184 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 0, leaderLastIndex: 1, leaderSnapShotIndex: -1 03:06:17,184 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): sendAppendEntries: 0 is present for follower follower 03:06:17,184 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Sending AppendEntries to follower follower: AppendEntries [leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=1, payload=1], SimpleReplicatedLogEntry [index=1, term=1, payload=2]]] 03:06:17,184 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): tryToCompleteLeadershipTransfer: followerId: follower, matchIndex: -1, lastIndex: 1, isVoting: true 03:06:17,386 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): sendAppendEntries: 0 is present for follower follower 03:06:17,386 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Sending AppendEntries to follower follower: AppendEntries [leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=1, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=1, payload=1], SimpleReplicatedLogEntry [index=1, term=1, payload=2]]] 03:06:17,386 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 1, nextIndex: 2 03:06:17,386 PM [main] [DEBUG] MockRaftActorContext - leader: fakeSnapshot purging log to 0 for term 1 03:06:17,386 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 03:06:17,387 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): tryToCompleteLeadershipTransfer: followerId: follower, matchIndex: 1, lastIndex: 1, isVoting: true 03:06:17,387 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Follower's log matches - sending ElectionTimeout 03:06:17,387 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 2, leaderLastIndex: 1, leaderSnapShotIndex: 0 03:06:17,387 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Leader transfer complete 03:06:17,389 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior272#-1435518502] 03:06:17,400 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader273] 03:06:17,400 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower274] 03:06:17,403 PM [main] [INFO] LeaderTest - Starting testInstallSnapshot 03:06:17,404 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from -1 to 1 03:06:17,404 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from 1 to 3 03:06:17,404 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Election: Leader has following peers: [follower] 03:06:17,404 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: 3, leaderLastIndex: 2, leaderSnapShotIndex: 2 03:06:17,404 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): sendInstallSnapshot 03:06:17,404 PM [main] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Snapshot 106 bytes, total chunks to send: 1 03:06:17,404 PM [main] [DEBUG] LeaderInstallSnapshotState - leader (Leader): Next chunk: total length=106, offset=0, size=106, hashCode=1072258452 03:06:17,404 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): next snapshot chunk size for follower follower: 106 03:06:17,404 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): InstallSnapshot sent to follower Vector(user, follower277), Chunk: 1/1 03:06:17,404 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior275#-1275054380] 03:06:17,415 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader276] 03:06:17,415 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower277] 03:06:17,422 PM [main] [INFO] LeaderTest - Starting testFollowerCatchUpWithAppendEntriesMaxDataSizeExceeded 03:06:17,422 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from -1 to 1 03:06:17,422 PM [main] [DEBUG] MockRaftActorContext - leader: Moving last applied index from 1 to 3 03:06:17,422 PM [main] [DEBUG] MockRaftActorContext - follower: Moving last applied index from -1 to 1 03:06:17,422 PM [main] [DEBUG] MockRaftActorContext - follower: Moving last applied index from 1 to -1 03:06:17,422 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Election: Leader has following peers: [follower] 03:06:17,422 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: false, followerNextIndex: 3, leaderLastIndex: 3, leaderSnapShotIndex: -1 03:06:17,423 PM [main] [INFO] MockRaftActorContext - follower (Follower): The followers log is empty and the senders prevLogIndex is 2 03:06:17,423 PM [main] [INFO] MockRaftActorContext - follower (Follower): Follower is out-of-sync so sending negative reply: AppendEntriesReply [term=1, success=false, followerId=follower, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:06:17,423 PM [main] [DEBUG] SyncStatusTracker - follower: Last sync leader does not match current leader leader, need to catch up to -1 03:06:17,423 PM [main] [INFO] MockRaftActorContext - leader (Leader): handleAppendEntriesReply - received unsuccessful reply: AppendEntriesReply [term=1, success=false, followerId=follower, logLastIndex=-1, logLastTerm=-1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5], leader snapshotIndex: -1, snapshotTerm: -1, replicatedToAllIndex: -1 03:06:17,423 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: -1, nextIndex: 0 03:06:17,423 PM [main] [INFO] MockRaftActorContext - leader (Leader): follower follower appears to be behind the leader from the last snapshot - updated: matchIndex: -1, nextIndex: 0 03:06:17,423 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 0, leaderLastIndex: 3, leaderSnapShotIndex: -1 03:06:17,423 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): sendAppendEntries: 0 is present for follower follower 03:06:17,423 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Sending AppendEntries to follower follower: AppendEntries [leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=3, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=1, payload=0], SimpleReplicatedLogEntry [index=1, term=1, payload=1]]] 03:06:17,423 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): handleAppendEntries: AppendEntries [leaderId=leader, prevLogIndex=-1, prevLogTerm=-1, leaderCommit=3, replicatedToAllIndex=-1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=0, term=1, payload=0], SimpleReplicatedLogEntry [index=1, term=1, payload=1]]] 03:06:17,423 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Number of entries to be appended = 2 03:06:17,423 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): After cleanup, lastIndex: -1, entries to be added from: 0 03:06:17,423 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Append entry to log 0 03:06:17,423 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Append entry to log 1 03:06:17,423 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Log size is now 2 03:06:17,424 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Commit index set to 1 03:06:17,424 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): handleAppendEntries returning : AppendEntriesReply [term=1, success=true, followerId=follower, logLastIndex=1, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:06:17,424 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 1, nextIndex: 2 03:06:17,424 PM [main] [DEBUG] MockRaftActorContext - leader: fakeSnapshot purging log to 1 for term 1 03:06:17,424 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 2, leaderLastIndex: 3, leaderSnapShotIndex: 1 03:06:17,424 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): sendAppendEntries: 2 is present for follower follower 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Sending AppendEntries to follower follower: AppendEntries [leaderId=leader, prevLogIndex=1, prevLogTerm=1, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=2, term=1, payload=2], SimpleReplicatedLogEntry [index=3, term=1, payload=3]]] 03:06:17,425 PM [main] [DEBUG] SyncStatusTracker - follower: Lagging 2 entries behind leader leader and reached 1 (of expected -1) 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): -1, lastIndex(): 1 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Setting last applied to 0 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower: Moving last applied index from -1 to 0 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Setting last applied to 1 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower: Moving last applied index from 0 to 1 03:06:17,425 PM [main] [WARN] MockRaftActorContext - follower (Follower): Missing index 2 from log. Cannot apply state. Ignoring 2 to 3 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): handleAppendEntries: AppendEntries [leaderId=leader, prevLogIndex=1, prevLogTerm=1, leaderCommit=3, replicatedToAllIndex=1, payloadVersion=5, recipientRaftVersion=5, leaderRaftVersion=5, leaderAddress=null, entries=[SimpleReplicatedLogEntry [index=2, term=1, payload=2], SimpleReplicatedLogEntry [index=3, term=1, payload=3]]] 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Number of entries to be appended = 2 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): After cleanup, lastIndex: 1, entries to be added from: 0 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Append entry to log 2 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Append entry to log 3 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Log size is now 4 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Commit index set to 3 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): handleAppendEntries returning : AppendEntriesReply [term=1, success=true, followerId=follower, logLastIndex=3, logLastTerm=1, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=5, raftVersion=5, recipientRaftVersion=5] 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): handleAppendEntriesReply - FollowerLogInformation for follower updated: matchIndex: 3, nextIndex: 4 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - leader: fakeSnapshot purging log to 2 for term 1 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - leader (Leader): Checking sendAppendEntries for follower follower: active: true, followerNextIndex: 4, leaderLastIndex: 3, leaderSnapShotIndex: 2 03:06:17,425 PM [main] [DEBUG] SyncStatusTracker - follower: Lagging 0 entries behind leader leader and reached 3 (of expected -1) 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): applyLogToStateMachine, appendEntries.getLeaderCommit(): 3, context.getLastApplied(): 1, lastIndex(): 3 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Setting last applied to 2 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower: Moving last applied index from 1 to 2 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower (Follower): Setting last applied to 3 03:06:17,425 PM [main] [DEBUG] MockRaftActorContext - follower: Moving last applied index from 2 to 3 03:06:17,426 PM [main] [DEBUG] MockRaftActorContext - follower: fakeSnapshot purging log to 1 for term 1 03:06:17,426 PM [main] [INFO] TestActorFactory - Killing actor Actor[pekko://test/user/behavior278#1166424825] 03:06:17,427 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/leader279] 03:06:17,427 PM [main] [INFO] TestActorFactory - Killing actor TestActor[pekko://test/user/follower280]