Late evening.
Replication was broken.
You had two terminal windows open.
One on the primary.
One on the replica.
Same prompt.
Same font.
Same black background.
The plan was simple.
Wipe the replica's data directory
and copy it fresh
from the primary.
You typed the command.
Pressed enter.
Then noticed
the hostname.
This is not a hypothetical.
On the last day of January 2017,
a GitLab engineer
did exactly this,
deleting data on the primary database
instead of the replica,
and the company lost
about six hours
of production data.
He stopped it
within seconds.
Seconds were enough.
Here is the mechanism.
Designers call it
a mode error.
The same action
means different things
depending on a state
you are not looking at.
The keys are identical.
The window is identical.
Only the mode
is different,
and the mode
lives in your memory
of which window
you clicked last.
Memory
is the first thing
a long evening
takes away.
Tiredness does not
make you type badly.
You type perfectly.
You just type
into the place
you believe you are,
and the belief
is an hour old.
Being careful
does not fix it,
because you were careful.
You were careful
in the wrong window.
So make the mode
impossible to miss.
Put the hostname
and the environment
in the prompt,
in capitals.
Make production
a different colour.
Red background,
if you can stand it.
Close the window
you are not using.
Two sessions open
on two servers
that look alike
is not efficiency.
It is a trap
with the lid already open.
Before anything destructive,
type the hostname
as part of the command,
or use a tool
that asks you for it.
On Debian and Ubuntu,
molly-guard does exactly that
for shutdown and reboot
over ssh.
And read the prompt
out loud
before you press enter.
It feels silly.
It is one second.
Your attention
is a single pointer.
Do not let it
point at the wrong server
just because
both servers
look like home.
– Serguey Asael Shinder