superfly/corrosion

Changes sent via sync goes way down on restart

Open

#129 opened on Dec 30, 2023

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Rust (83 forks)auto 404
:detective: investigative workhelp wanted

Repository metrics

Stars
 (1,817 stars)
PR merge metrics
 (PR metrics pending)

Description

On restart, the metric for sent changes via sync shows a marked decrease. This may only happen if restarting a few nodes.

It appears the node's sent sync only recover after it logs a "rejoined cluster" message. I'm not sure how this can happen and it should only happen when things are "wrong" in the SWIM cluster. I don't see any logs indicating that's the case though.

The node should've been announcing a new identity on start-up anyway, so I'm not sure how this rejoin works out.

Perhaps we need to tweak the config for the foca cluster.

Contributor guide

Research direction
Investigate why the 'sent changes via sync' metric drops on restart and how the 'rejoined cluster' message relates to recovery. Look into the SWIM gossip protocol implementation and node identity handling on restart.
Tech stack
rust
Domain
backenddistributed systemsnetworking
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
RustDistributed Systems
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65