Secure and Fault-Tolerant Replicated Control Plane

The distributed implementation currently relies on a central distributor or broker to coordinate workers. This creates a single point of failure: if that process becomes unavailable, the system may no longer be able to allocate work, track workers, or control execution. See this link to try an interactive explanation. Inah's lecture slides are here.

Visual guide: what is changing

B1 to B4 mark trust boundaries: places where data crosses between parties that should not automatically trust each other. The cluster agrees on the authoritative state (leader, workers, job and generation, allocation, commands, latest valid checkpoint), so losing one replica does not lose the state, and a surviving majority can carry on.

Replace the single broker with a small replicated control-plane cluster based on the Raft consensus algorithm. The cluster should maintain authoritative state for:

• the current control-plane leader;
• registered worker nodes;
• the current Game of Life job and generation;
• work allocation or scheduling information;
• pause, resume, checkpoint, and termination commands; and
• the location and generation number of the latest valid checkpoint.

Visual guide: try a Raft cluster

A deliberately simplified model for building intuition. Crash nodes, cut them off from the network, send commands, and watch terms, elections and the replicated log. It is not a Raft implementation and leaves out timers, randomised timeouts and many details.

Cluster size 3 replicas5 replicasQuorum = 2 of 3 · 3 reachable now · can make progress

Node 1leader

term 1 · committed 1

∅

Node 2follower

term 1 · committed 1

∅

Node 3follower

term 1 · committed 1

∅

committed entry not committed (must not be acted on) leader no-op

• Cluster started: node 1 won the first election (term 1).

Things to try

1. Crash the leader. Who wins, and what happens to the term?
2. Cut off the leader (it keeps running). Send a command to the old leader, then to the new one. Reconnect the old leader. What happens to its entry?
3. Crash a node, send several commands, restart it. Can it become leader straight away? Why or why not?
4. Crash a majority. What can and can't the cluster still do? Is that the behaviour you want for a control plane?
5. Compare 3 and 5 replicas: how many failures does each tolerate, and what does each extra replica cost you?

Your system should continue to operate, or recover predictably, when the current leader fails. A remaining replica should be elected as leader and resume coordination without allowing conflicting control decisions or silently reverting to stale state.

Add suitable security controls to protect communication with the control plane. Your design should address:

• authentication of control-plane replicas and workers;
• prevention of unauthorised cluster membership;
• integrity protection for control messages;
• detection of duplicated, delayed, or replayed commands;
• rejection of commands from a stale former leader; and
• detection of corrupted or tampered checkpoint metadata.

Visual guide: where each security requirement bites

Pick a requirement to see where in the architecture it applies, and some questions your design and threat model should answer. These are prompts, not answers.

Authenticate replicas and workers

• Who has to prove who they are to whom, in each direction?
• How are identities created, distributed and revoked?
• What happens to a node whose identity is revoked while it is running?

Before implementing the extension, produce a lightweight threat model containing:

• the assets that require protection;
• the main trust boundaries;
• plausible threats and failure scenarios; and
• security requirements derived from those threats.

Visual guide: the threat model as a chain of evidence

1 · AssetsWhat would hurt if it were lost, altered or leaked?

→

2 · Trust boundariesWhere does data cross between parties with different levels of trust?

→

3 · ThreatsWhat could go wrong at each crossing, and what failures are plausible?

→

4 · RequirementsOne testable statement per threat that you intend to address.

→

5 · Controls and testsThe mechanism you built, and the experiment that shows it works.

Aim for traceability in both directions: every requirement should come from a threat, and every claim in your evaluation should point back to a requirement. The brief also asks you to say which threats are out of scope; a threat model that shows what you chose not to defend against, and why, is stronger than one that claims everything.

Conduct experiments to demonstrate the behaviour of your system under at least the following conditions:

• failure of a follower replica;
• failure of the current leader;
• restart of a failed replica;
• delayed or duplicated control messages;
• an attempted connection by an unauthorised node; and
• stale or modified checkpoint metadata.

Visual guide: what each experiment should demonstrate

Condition	The question it answers	Evidence to think about collecting
Follower fails	Does the cluster keep making decisions with one replica fewer?	Commands still commit; generation progress; any change in message traffic
Leader fails	Is a new leader elected, and does nothing already committed get lost or contradicted?	Election time; the job's generation before and after; no duplicated or skipped control decisions
Failed replica restarts	Does it rejoin safely and catch up without disturbing the current leader?	Its state before and after; time to catch up; confirmation it did not become a competing leader
Delayed or duplicated messages	Is a repeated or late command acted on at most once, or rejected?	What the receiver logs; the final state compared with a run without interference
Unauthorised node connects	Is it refused, and how do you know the cluster membership did not change?	Rejection evidence on both sides; membership view before and after
Stale or modified checkpoint metadata	Is it detected before it is used to restore the job?	The check failing; what the system does next; a control case with valid metadata

Measure the performance overhead introduced by replication and the security controls. Compare the extended implementation with the original single-broker design. You may consider:

• total execution time;
• control-message traffic;
• leader-election time;
• recovery time; and
• the effect of failures on completed generations.

Visual guide: defining your timing metrics

These are suggested definitions. Your report should state exactly when each timestamp is taken (and by which process) so that your numbers can be reproduced. Consider also what happens to generations if the job has to resume from a checkpoint: are any generations repeated, and how do you show the result is still correct?

In your report:

• explain the architecture and replicated state model;
• present your threat model and derived security requirements;
• explain how leader election, state replication, authentication, and recovery operate;
• provide evidence that control decisions remain consistent during the tested failures;
• evaluate which threats your implementation addresses and which remain outside its scope; and
• analyse the trade-offs between availability, consistency, security, performance, and implementation complexity.

Visual guide: a suggested path through the work

1. Threat model firstAssets, boundaries, threats, requirements (the brief asks for this before implementing).
2. Decide the architectureHow many replicas, what is replicated, how workers find the leader, and whether you build consensus or adopt an implementation (justify either).
3. Replicate state, no security yetGet leader election and a replicated log working with a plain connection. Test failover.
4. Move the job onto the clusterRegistration, allocation, pause, resume, checkpoint and terminate all go through the leader and the log.
5. Add identity and integrityAuthenticate replicas and workers; refuse unknown nodes; protect control messages.
6. Add freshness protectionsHandle duplicated, delayed or replayed commands and reject a stale former leader.
7. Protect checkpoint metadataMake tampering or staleness detectable before restore.
8. Run the six experimentsAgainst the original single-broker baseline, with metrics defined up front.
9. Write the evaluationIncluding what remains out of scope and the trade-offs.

Report map

Report bullet	Evidence to bring
Architecture and replicated state model	An architecture diagram with trust boundaries; a table of what state is replicated
Threat model and security requirements	Assets, boundaries, threats, requirements, with traceability
How election, replication, authentication and recovery operate	Sequence diagrams for a normal command, a failover, and a rejected node
Control decisions stay consistent during failures	Experiment logs and state comparisons from the six conditions
Threats addressed and not addressed	A table linking each threat to a control and a test, or to an explicit scope decision
Trade-offs	Overhead measurements against the baseline, and discussion of availability, consistency, security, performance and complexity