Redis latency benchmark: HotKV vs Valkey and Dragonfly

This Redis latency benchmark measures how long HotKV, Redis, Valkey and Dragonfly take to answer at a fixed request rate. Our earlier benchmarks measured throughput, and a reader rightly pointed out that throughput is not everything. So we held the load constant, raised it step by step, and recorded the median (p50) and the 99th percentile (p99) at each step. We also measured pub/sub delivery time.

In short. In this Redis latency benchmark on AWS, HotKV and Dragonfly both held 1.6 million GETs per second, Redis with 8 I/O threads held about 0.8 million, Valkey with 8 I/O threads about 0.4 million, and single-thread Redis and Valkey about 0.2 million. From 200,000 to 1.6 million requests per second HotKV's p99 stayed between 0.59 and 0.76 ms, against 0.71 to 1.48 ms for Dragonfly. At light load the multi-threaded products tied on p99, and pub/sub delivery was a tie.

Why we ran a Redis latency benchmark at a fixed rate

A throughput test pushes a server as hard as it will go and reports how much it served. That says little about how long one request waits. A latency test fixes how many requests per second the clients send, and measures the time each takes. The server is then at the same load as every other product, so the times can be compared.

This kind of test has a catch. memtier_benchmark is a closed-loop tool, so a client waits for a reply before it sends the next request. If a server cannot keep up, the client simply sends fewer requests, and the latency of the ones it did send looks better than it should. We therefore count a step as sustained only when the achieved rate is at least 97 percent of the rate we asked for. Steps that fail the test are marked "not sustained" in the tables and their latency is not shown.

What we tested

Machines

• Servers and load generator. c8a.12xlarge instances (AMD EPYC 9R45, 48 cores, no SMT, 96 GiB, 22.5 Gbit/s network) in one AWS cluster placement group in eu-south-2a, running Ubuntu 24.04. One instance ran the load generator and one ran the server.
• One product at a time. Each product ran alone on the server, so nothing overlapped and nothing shared the network. Every server process was pinned to CPUs 0 to 39, with persistence off and protected mode off.

Products

• HotKV 0.3.0 with 40 threads.
• Dragonfly 2.0.2 with 40 proactor threads.
• Redis 8.10.3 and Valkey 9.1.3, each run twice, once in the default single-thread setup and once with --io-threads 8.

The launch line and the live CONFIG GET output of every run are saved with its results.

Workload and procedure

The data set was 1,000,000 keys with 256-byte values, and pipeline depth was 1, so each request waits for its own reply. The load was GET only, and a mix of 1 write to 10 reads. Each step ran for 20 seconds, twice, and we report the mean of the two runs. memtier_benchmark used a per-connection rate limit and recorded an HDR histogram. We ran two connection counts.

• 816 connections (24 threads of 34 connections), for rates from 20,000 to 2.4 million requests per second.
• 48 connections (4 threads of 12 connections), for rates up to 300,000 requests per second. Above that, 48 connections cannot reach the rate, because each connection can only send as fast as it gets replies.

Results

The GET tables come first, one for each connection count, followed by pub/sub.

At 816 connections

Each cell is p50 / p99 in milliseconds, for GET. "NS" means not sustained, and the rate it did reach is in brackets. A dash means the sweep for that product had already stopped.

The mixed workload gave the same picture. HotKV's p99 was 0.54 to 0.75 ms from 200,000 to 1.6 million requests per second, and the points each product could sustain were the same as for GET. All of it is in the results folder.

How to read the single-thread rows

The 1.6 to 3 ms medians of single-thread Redis and Valkey at 816 connections are not their steady-state latency. With a rate limit, memtier_benchmark makes all 816 connections fire at the same moment. A single-thread server has to work through that burst of 816 requests one after another, so the median of a request includes its place in the queue. We ran the 48-connection test below to remove this effect.

At 48 connections

The cells have the same meaning as above, for GET only.

Pub/sub delivery time

We measured the time from a publish to its delivery to a subscriber, with 64 publishers, 100 subscribers and a 128-byte payload, using pubsub-sub-bench. Each message went to one subscriber. Cells are p50 / p99 in milliseconds.

All products delivered in the same range. At the highest load the single-thread products had a higher median, 0.51 and 0.57 ms against 0.11 ms for HotKV.

We also had to patch pubsub-sub-bench. It compared a publish time in milliseconds with a receive time in microseconds. The patch is in the results folder.

What the results show

The tables show a lead for HotKV and Dragonfly at high load, a tie at light load, and a ceiling for single-thread Redis and Valkey.

HotKV and Dragonfly kept a low tail to 1.6 million requests per second

Both sustained 1.6 million GETs per second at 816 connections, with a median of 0.28 ms. HotKV's p99 was between 0.59 and 0.76 ms from 200,000 requests per second up. Dragonfly's was between 0.71 and 1.48 ms, and the two were close at 1.6 million (0.64 against 0.71 ms). Redis and Valkey with I/O threads stopped sustaining the rate at lower loads, and single-thread Redis and Valkey stopped at about 260,000.

At light load the multi-threaded products tie

At 48 connections, up to 300,000 requests per second, HotKV, Dragonfly and Redis with 8 I/O threads all had a p99 of about 0.27 ms. HotKV had the lowest median, 0.08 to 0.09 ms from 100,000 requests per second, with Dragonfly at 0.08 to 0.10 ms. A difference of 0.01 ms is within what other parts of the setup can cause, so we do not claim it as a win.

Single-thread Redis and Valkey ran out at about 230,000 a second

At 48 connections they had a median of about 0.2 ms and a p99 of 0.35 to 0.43 ms, and they did not sustain 300,000 requests per second (about 231,000 and 233,000). That is the ceiling of one thread on this hardware and workload. It is why Redis and Valkey are usually run as many processes or as a cluster, as in our cluster benchmark.

Limitations

These are HotKV Ltd's measurements of the versions and configuration above. They are not a claim about any other version, configuration or workload of these products.

The test setup

• One server, one load generator. Two repeats per step, on one set of machines on one date. We did not measure variation between machines or days.
• Layouts we chose. Redis and Valkey ran in their default single-thread setup and with --io-threads 8, and we did not tune either beyond that. Other settings could change their figures. If you think a setting here disadvantages a product, tell us and we will check it and publish the result.
• The tool's burst effect. At 816 connections, the medians of single-thread products are inflated by the way memtier_benchmark rate limiting works, as explained above.

What was not measured

• Data. Larger or skewed values, memory per key, and persistence. Persistence was off for every server.
• Modes. Cluster mode, sharded pub/sub, TLS, replication and failover.
• Pub/sub fan-out. One message to many subscribers.

Redis, Valkey and Dragonfly are trademarks of their respective owners. They are named only to identify the software we measured, and this implies no sponsorship, endorsement or affiliation.

Reproduce it

The sweep scripts, the launch line and live configuration of every server, the pubsub-sub-bench patch and every raw result are in the benchmark repository. The latency script is harness/lat_sweep.sh, and the pub/sub script is harness/ps_sweep.sh. The raw folder holds one archive per test with the memtier_benchmark JSON, the HDR histograms and the CPU use of the server for every run.

Try it yourself

HotKV Cloud runs HotKV for you. Start on HotKV Cloud, or talk to sales to benchmark HotKV against your own workload.

Frequently asked questions

Is HotKV faster than Redis in latency?

Under load, at the rates we tested, yes. At 816 connections and 200,000 requests per second HotKV had a p99 of 0.59 ms, Redis with 8 I/O threads 1.11 ms and single-thread Redis 5.60 ms. At light load with 48 connections the multi-threaded products tied on p99, at about 0.27 ms.

Why did you hold the request rate fixed?

At a fixed rate every product carries the same load, so the latencies can be compared. A throughput test shows how much a server serves at its limit, which says little about how long one request waits.

What does "not sustained" mean?

The product achieved less than 97 percent of the request rate we asked for. memtier_benchmark is a closed-loop tool, so an overloaded server makes the clients send less and the latency of the remaining requests looks better than it is. We show no latency for those steps.

How did pub/sub delivery time compare?

At the three loads we tested, all products delivered in the same range, with a median of about 0.1 ms at the two lower loads. The table above has every figure.

Can I reproduce this Redis latency benchmark?

Yes. The scripts, the configurations and every raw result are in the benchmark repository. You need two servers on the same network, a load generator, and the four products.