How Others Test Cross-Zone Locally

tc netem, Toxiproxy, Pumba, Chaos Mesh, LitmusChaos, zone labels in kind and k3d, Jepsen and Testcontainers: what each gives, a short config form for each, and an honest comparison with what we built.

👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

Every claim about a tool below links to its own documentation, read for this article. Our side of the comparison is what part 3 built; our failure-scenario runs are still planned.

This is part 5 of the series on testing cross-zone, cross-datacenter and cross-provider behaviour on a laptop. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.

Five layers

Part 4, Placement In Every Trace, made every span say where it ran. Before going further with our own emulation, it is worth looking at what the world already uses. The tools sort into layers by where they act:

The lower the layer, the more it can do and the less it knows about your system. The higher, the more it knows and the more it needs installed.

tc netem

Everything network-shaped in this list ends at the kernel's netem sooner or later. It adds delay, jitter and loss to packets leaving a device (tc-netem(8)):

tc qdisc change dev eth0 root netem delay 100ms 20ms distribution normal tc qdisc change dev eth0 root netem loss 0.3% 25% 

The man page has one warning worth repeating: for TCP performance tests, netem should sit on the ingress of the receiver, not on the sender. On its own, netem knows nothing about which peer is which; that is up to the filters around it.

Toxiproxy

Toxiproxy is "a TCP proxy to simulate network and system conditions for chaos and resiliency testing" (Toxiproxy). The client connects to the proxy instead of the real service, and toxics are added to the proxy over an HTTP API on port 8474:

toxiproxy-cli create -l localhost:26379 -u localhost:6379 redis toxiproxy-cli toxic add -t latency -a latency=1000 redis 

The toxics are latency, down, bandwidth, slow_close, timeout, reset_peer, slicer, limit_data and packet_loss. That is a richer vocabulary than netem has, at one price: the client must be pointed at the proxy.

Pumba

Pumba does chaos and network emulation for containers on Docker, containerd and Podman (Pumba). It runs tc from a helper that joins the target's namespace, so the target image needs nothing:

pumba netem --duration 5m delay --time 3000 mydb pumba iptables --duration 2m loss --probability 0.1 myapp 

That is the same trick as our sidecar in part 3, packaged as a tool.

Chaos Mesh and LitmusChaos

In Kubernetes, network chaos becomes a resource. Chaos Mesh's NetworkChaos supports netem, delay, loss, duplicate, corrupt, partition and bandwidth, selects pods by label and has a direction of to, from or both (Chaos Mesh):

apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: { name: delay-example } spec: action: delay mode: one selector: { namespaces: [default], labelSelectors: { app: frontend } } delay: { latency: '20ms', jitter: '5ms', correlation: '0' } 

LitmusChaos, a CNCF Incubating project, describes itself as "a Cloud-Native Chaos Engineering Framework with cross-cloud support" (Litmus). Its pod-network-latency experiment starts tc with netem inside the target, takes NETWORK_LATENCY, JITTER and DESTINATION_IPS or DESTINATION_HOSTS to aim at one peer, and can use Pumba as its library (pod-network-latency).

Zones in kind and k3d

Zones on a laptop cluster are node labels. kind takes them in the cluster config (kind); k3d takes them on the command line with a node filter (k3d):

# kind nodes: - role: worker labels: { topology.kubernetes.io/zone: a } - role: worker labels: { topology.kubernetes.io/zone: b } 

# k3d k3d cluster create --agents 3 \ --k3s-node-label "topology.kubernetes.io/zone=a@agent:0" \ --k3s-node-label "topology.kubernetes.io/zone=b@agent:1" \ --k3s-node-label "topology.kubernetes.io/zone=c@agent:2" 

Then topologySpreadConstraints spread the pods. DoNotSchedule tells the scheduler not to place a pod that would break the skew; ScheduleAnyway places it anyway, preferring nodes that reduce the skew (Kubernetes):

topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: { matchLabels: { app: my-app } } 

Labels and spread give zones as a scheduling fact. They give no distance: two "zones" on a laptop are as close as two processes.

Jepsen

Jepsen is at a different layer altogether. It "aims to improve the safety of distributed databases, queues, consensus systems" (Jepsen). While a test runs operations against a system, a nemesis process introduces faults, and a checker then analyses the recorded history for correctness (jepsen-io/jepsen). Network manipulation in its tests is typically done with iptables.

What Jepsen adds is not a better way to cut a network. It is a rigorous answer to "was the result correct", which every tool above leaves to you.

Testcontainers

For tests in code, Testcontainers wraps Toxiproxy as a module. A ToxiproxyContainer joins the same network as the service under test, a proxy is created in front of it, and toxics are added from the test (Testcontainers):

proxy.toxics().latency("latency", ToxicDirection.DOWNSTREAM, 1_100).setJitter(100); proxy.toxics().bandwidth("CUT_CONNECTION_DOWNSTREAM", ToxicDirection.DOWNSTREAM, 0); proxy.toxics().bandwidth("CUT_CONNECTION_UPSTREAM", ToxicDirection.UPSTREAM, 0); 

A connection cut is bandwidth zero in both directions; removing the toxics restores it.

Side by side

The honest summary: we did not invent a new way to delay a packet. Under the hood it is netem and iptables, like most of the list. What is ours is the model above them - one declaration of a planet, from which every rule, every placement and every chaos action is derived. None of the tools above has a continent or a provider as a concept.

What we would borrow

Two things, both from the top of the list. Toxiproxy's vocabulary - slow_close, reset_peer, limit_data - names failures that netem cannot express. And Jepsen's stance: a chaos run without a strict check of the result is a demonstration, not a test. Both are ideas, not plans yet.

What I do not claim

• Not a ranking of these tools. Each is good at its own layer.
• Not that our emulation does what Jepsen does. It does not check histories.
• Not that any of our scenarios has run. They are planned.
• Not completeness. These are the tools I read for this part, not all of them.

What of this is method

• Know the layer of a tool. Kernel, proxy, container, cluster, history - each sees a different system.
• Read the documentation, not the reputation. Every claim here has a link.
• A cut is not a test. The check of the result is the test.

Cross-anything on a laptop - Part 5. Next: Every Verdict Carries A Trace.

If you use Toxiproxy in tests, which toxic found the most? If you run Chaos Mesh or Litmus, do you run it locally or only on a shared cluster? If you have run Jepsen, what did the checker catch that nothing else would? Which of these would you add to a laptop first?