STATERA Library · Series 08 · Volume 08 · Chapter 10 · EN
Source: https://github.com/Statera-Library/08_Vol_08_Robot_Database_and_Storage/blob/main/Chapter_10_Database_Replication_and_HA/Chapter_10_Database_Replication_and_HA_EN.md
Volume 08 Robot Database and Storage
10. Database Replication and HA
10.01 DB HA Patterns: Active.Passive / Active.Active
High availability in database systems is the architectural discipline of keeping data services accessible when servers, storage devices, networks, operating systems, or database processes fail. In robotics, this requirement can be especially important because fleet management, mission scheduling, robot configuration, telemetry, maps, and operational state may depend continuously on database services. A high-availability design therefore combines redundancy, replication, failure detection, traffic routing, and controlled recovery rather than relying on a single powerful database server.
Database high availability is commonly described through active-passive and active-active patterns. Both maintain more than one database instance, but they differ in how those instances process workloads. Active-passive architectures normally designate one database node as the operational primary while another remains synchronized and prepared to take over. Active-active architectures allow multiple nodes or sites to process workloads simultaneously, requiring additional mechanisms for coordinating writes, replication, consistency, and conflict resolution.
In an active-passive architecture, applications send normal database traffic to the active node. The passive node receives replicated data but generally does not accept application writes while operating as a standby. Replication may be synchronous, asynchronous, or a combination depending on durability and latency requirements. When the active node fails, the standby is promoted and client connections are redirected to it, restoring database service without requiring the failed machine to return immediately.
The major advantage of active-passive design is architectural clarity. Because one node normally owns the writable database state, application developers face fewer concurrent-write conflicts and the system can preserve familiar transactional behavior. Operations teams can also reason more easily about primary ownership, replication state, and failover. The main disadvantage is that standby resources may provide limited productive capacity during normal operation even though they consume infrastructure, maintenance, monitoring, and licensing resources.
Failover is the central operational process in active-passive HA. A monitoring component detects that the active database is unavailable or unhealthy, verifies whether promotion is safe, selects an eligible standby, promotes it, and redirects database clients. This sequence must distinguish genuine failures from temporary network interruptions. An incorrect promotion can create two nodes that both believe they are primary, producing the dangerous split-brain condition addressed later in this chapter structure.
Synchronous replication can reduce potential data loss by requiring a replica to acknowledge selected changes before a transaction is considered committed. This strengthens durability across node failure but adds communication latency to transaction processing. Asynchronous replication allows the primary to commit without waiting for the replica, usually improving response time but creating replication lag. If the primary fails before outstanding changes reach the standby, recently committed transactions may be unavailable after promotion.
Active-active architecture changes the model by allowing two or more database nodes, clusters, or geographic sites to serve application workloads concurrently. Instead of reserving one environment mainly for failure recovery, infrastructure can contribute to normal service capacity. Requests may be distributed by geography, tenant, robot fleet, shard, or application function. If one active location fails, traffic can potentially move toward another location that was already processing live workloads.
The difficulty of active-active operation lies primarily in data coordination. When multiple nodes can modify the same logical information, concurrent updates may occur before replication delivers each change everywhere. The database architecture must therefore define ownership, consensus, conflict detection, conflict resolution, or application-level reconciliation. Some designs avoid conflicts by assigning specific records or partitions to particular writers, while distributed databases may use quorum or consensus mechanisms to coordinate state across nodes.
Consistency requirements should determine replication architecture rather than treating active-active as automatically superior. Strongly consistent transactions may require coordination among nodes before accepting a write, which can increase latency as network distance grows. Systems prioritizing availability and low-latency local operations may permit temporary divergence and converge later. Robot systems often contain both categories: mission ownership and safety-relevant configuration may demand strict consistency, while some telemetry or historical observations can tolerate delayed synchronization.
Geographic distribution makes this tradeoff more visible. A robot fleet operating across several factories, warehouses, or cities may benefit from databases positioned close to each operational site. Local applications can continue using nearby services when a remote region becomes unreachable. However, inter-region latency and network partitions prevent designers from assuming that every location can synchronously coordinate every transaction at negligible cost. Data domains must therefore be classified according to consistency, latency, and recovery requirements.
High availability should also be distinguished from backup and disaster recovery. Replication creates another current or nearly current copy of database state, but replication can also reproduce accidental deletions, corrupted records, or malicious modifications. Backup retains recoverable historical states, while HA focuses primarily on service continuity. A robust robot data platform consequently combines replication and failover with backup, point-in-time recovery, integrity verification, and disaster-recovery procedures rather than replacing one mechanism with another.
Recovery objectives provide measurable requirements for this architecture. Recovery Time Objective expresses how long a database service may remain unavailable, while Recovery Point Objective expresses how much recent data loss can be tolerated. These targets influence whether manual failover is sufficient, whether automatic promotion is required, how many replicas should exist, and whether synchronous replication is justified. They also connect HA design directly with the backup and recovery concepts developed in the preceding chapter.
Robot databases should not assign identical HA requirements to every dataset. A fleet scheduler, robot authorization database, mission state store, or operational configuration service may require rapid recovery because interruption can stop multiple robots. Historical logs and large sensor archives may accept longer restoration periods. Separating these service classes prevents organizations from applying expensive synchronous redundancy universally while still protecting database functions that directly affect ongoing robot operations.
Client connectivity is another essential layer. Promoting a standby accomplishes little if applications continue connecting to the failed address. HA systems therefore commonly use virtual endpoints, proxies, service discovery, cluster-aware drivers, load balancers, or orchestrators to direct applications toward healthy database instances. Connection pools must also recognize broken sessions and reconnect safely. Robot applications should be designed for transient database failures instead of assuming that every connection remains permanently valid.
Failover must preserve application semantics as well as connectivity. A transaction may have reached the database immediately before a connection was lost, leaving the client uncertain whether the operation committed. Blindly repeating such requests can create duplicate mission records, commands, or events. Idempotent operations, transaction identifiers, unique constraints, retry policies, and explicit state verification help applications recover safely. Database HA is therefore partly an application architecture problem rather than purely an infrastructure configuration problem.
Monitoring determines whether an HA design works reliably in production. Operators should observe node health, replication delay, transaction throughput, storage capacity, connection counts, error rates, replication state, and failover events. Alerting should identify degradation before redundancy disappears completely. A primary database that remains operational while its only replica has been disconnected for hours is technically available, but the system has already lost the redundancy required to survive the next failure.
HA testing must intentionally exercise failure conditions. Teams should test database-process termination, server loss, storage faults, network partitions, replica lag, exhausted disk capacity, and loss of communication between cluster members. Tests should measure detection time, promotion time, application reconnection, transaction behavior, and data consistency after recovery. Periodic exercises reveal assumptions that normal monitoring cannot expose and prepare operators to respond predictably when real infrastructure failures occur.
For robot fleets, database availability should be considered together with edge autonomy. Robots should not become immediately unsafe merely because a central database connection disappears. Essential local control, collision avoidance, and short-term mission execution can remain at the robot or edge layer while centralized databases maintain fleet-level state and coordination. This separation reduces the consequence of temporary database outages while HA mechanisms restore centralized services and reconcile accumulated operational information afterward.
The choice between active-passive and active-active therefore depends on workload characteristics rather than terminology. Active-passive is often appropriate when transactional simplicity, deterministic ownership, and straightforward failover are priorities. Active-active becomes attractive when multiple locations must serve traffic concurrently, geographic latency matters, or service capacity should remain distributed during failures. Its benefits must be balanced against greater complexity in synchronization, consistency management, routing, conflict handling, and operational diagnosis.
A mature architecture may combine both patterns. A regional robot platform could operate active-active services across major sites while each regional database cluster internally uses an active-passive or consensus-based redundancy mechanism. Data can also be partitioned so that each region owns particular robot fleets while replicating selected information globally. Such layered designs demonstrate that database HA is not a single configuration but a hierarchy of failure domains, replication relationships, and application responsibilities.
Ultimately, database high availability is achieved when infrastructure failure becomes an anticipated operating condition rather than an exceptional event. Redundant nodes, appropriate replication modes, reliable health detection, controlled failover, safe client retries, continuous monitoring, and repeated failure testing must work together. These principles provide the conceptual foundation for the subsequent PostgreSQL, Patroni, MySQL or MariaDB, Redis, Elasticsearch, geo-distributed replication, monitoring, and split-brain topics defined in this chapter.
10.02 PostgreSQL Streaming Replication Deep Dive [w/Code]
PostgreSQL streaming replication is a physical replication mechanism that continuously transfers Write-Ahead Log (WAL) records from a primary database server to one or more standby servers. Instead of reproducing individual SQL statements, the standby applies low-level changes recorded in WAL, allowing it to maintain a binary-compatible copy of the primary database cluster. This mechanism forms a fundamental building block for PostgreSQL high availability, read scaling, and disaster recovery architectures.
Write-Ahead Logging is the foundation of PostgreSQL replication. Before modified database pages are permanently written to data files, PostgreSQL records the corresponding changes in WAL. This ordering supports crash recovery because committed changes can be reconstructed from the log after a failure. Streaming replication extends the same principle across servers by transmitting WAL records to standby systems, where PostgreSQL replays them to reproduce the primary server\'s database state.
The primary server generates WAL continuously as transactions insert, update, delete, or modify database objects. A WAL sender process on the primary communicates with a WAL receiver process running on each connected standby. The receiver obtains WAL records through PostgreSQL\'s replication protocol and writes them locally. A startup process on the standby then replays those records against its database files, progressively advancing the standby toward the current state of the primary.
A standby normally begins from a consistent physical copy of the primary database cluster. PostgreSQL tools such as pg_basebackup can create this initial base backup while the primary remains operational. Once the required database files and replication configuration are available, the standby starts recovery and connects to the primary. Streaming replication then supplies WAL generated after the base backup, allowing the standby to catch up without repeatedly copying the entire database.
Physical replication requires compatibility between primary and standby environments because WAL describes changes at PostgreSQL\'s storage level rather than as portable logical operations. The standby represents the database cluster as a whole rather than selectively subscribing to arbitrary tables. This differs from logical replication, which operates on logical data changes and can replicate selected tables. Streaming replication is therefore particularly suitable when the objective is maintaining a complete standby for availability and recovery.
PostgreSQL supports asynchronous streaming replication as the common low-latency operating model. In this configuration, the primary can report a transaction as committed without waiting for the standby to confirm that the corresponding WAL has been received or persisted. Application response time is therefore less dependent on replica communication. However, if the primary fails while some WAL remains in transit, the promoted standby may not contain the most recently committed transactions.
Replication lag describes the distance between the primary\'s current WAL position and the progress achieved by a standby. Lag may arise from limited network bandwidth, storage latency, high transaction volume, insufficient standby resources, long recovery operations, or temporary connectivity loss. A replica that is technically connected but significantly behind the primary may provide weaker recovery characteristics than expected. Monitoring replication progress is consequently essential when streaming replication supports production HA.
Synchronous replication changes the commit path by requiring confirmation from designated standby systems according to the configured synchronization policy. A transaction on the primary may wait until the required replica has received, written, flushed, or applied the relevant WAL, depending on configuration. This can substantially reduce the risk of committed transaction loss during primary failure, but database latency becomes more sensitive to network conditions and standby performance.
The choice between synchronous and asynchronous replication should reflect Recovery Point Objective (RPO), transaction criticality, network characteristics, and acceptable application latency. A mission-state database for a robot fleet may justify stronger durability guarantees than a database containing reconstructable operational statistics. Conversely, forcing geographically distant replicas into every synchronous commit path can increase transaction latency and make remote infrastructure conditions directly visible to applications.
PostgreSQL provides replication slots to help coordinate WAL retention for replicas and other replication consumers. A physical replication slot records the WAL position still required by a standby, preventing the primary from removing necessary WAL prematurely. This improves resilience when a replica disconnects temporarily. However, an inactive slot can cause WAL files to accumulate on the primary, potentially exhausting storage, so slot status and retained WAL volume must be continuously monitored.
WAL retention can also be supported through configuration and WAL archiving. Streaming replication delivers records directly while the connection is available, whereas archived WAL provides another path for retrieving historical log segments. A standby can therefore combine restored archive files with streamed WAL. This combination is valuable for resilient recovery architectures because temporary streaming interruptions do not necessarily require rebuilding the replica when the missing WAL remains available from an archive.
Hot standby capability allows read-only queries to execute while a standby continues replaying WAL. This makes replicas useful for reporting, monitoring, analytics, or other read workloads rather than reserving all standby resources exclusively for failure recovery. Read scaling must nevertheless account for replication delay because queries executed on a standby may observe an older database state than queries executed on the primary, especially when asynchronous replication is used.
Read workloads can also interact with WAL replay. A long-running query on the standby may require row versions that incoming WAL needs to remove or modify as part of recovery. PostgreSQL must then resolve the conflict, potentially delaying replay or canceling the conflicting standby query depending on configuration and workload behavior. HA replicas used simultaneously for analytical workloads should therefore be designed so that reporting activity does not undermine replication freshness or recovery readiness.
When a primary database becomes unavailable, a sufficiently current standby can be promoted to become a new writable primary. Promotion ends recovery mode and allows normal read-write transactions. Streaming replication itself provides the replicated database state required for this transition, but complete high availability also requires failure detection, promotion policy, client redirection, and protection against multiple primaries. These responsibilities are commonly handled by additional orchestration components.
After promotion, the old primary cannot simply resume normal writable operation without considering the new cluster history. If both systems accept writes independently, their timelines diverge and split-brain can occur. PostgreSQL uses timeline histories to represent changes in recovery lineage following promotion. Administrators or HA automation must ensure that the former primary is isolated, repaired, rewound when appropriate, or rebuilt before it rejoins the replication topology.
pg_rewind can be useful when a former primary has diverged from the newly promoted server but remains sufficiently compatible for synchronization. Rather than performing another complete base backup, pg_rewind identifies changed blocks and adjusts the old server to follow the new timeline. This can reduce recovery time and network transfer for large database clusters, although correct prerequisites, WAL availability, and operational procedures remain essential before the server is safely returned as a standby.
Cascading replication allows a standby to provide WAL to additional downstream standbys. Instead of every replica connecting directly to the primary, intermediate replicas distribute replication traffic. This architecture can reduce network and WAL-sender load on the primary and can be useful across sites or network boundaries. The trade-off is that downstream replicas inherit additional replication paths and may experience greater lag or become disconnected when an upstream standby fails.
For robot and Physical AI systems, streaming replication can protect operational databases containing fleet state, mission records, robot configuration, user authorization, maintenance information, and structured telemetry metadata. A local standby can provide rapid recovery from server failure, while additional replicas may support read-intensive fleet dashboards or remote recovery. Large raw sensor datasets, video, point clouds, and training archives are normally better handled by storage architectures designed for those data types.
Replication health should be monitored through PostgreSQL statistics and operational metrics rather than inferred only from whether a standby process is running. Important observations include replica connection state, WAL sender and receiver activity, WAL positions, replay progress, replication lag, replication-slot status, retained WAL volume, transaction latency, storage consumption, and standby recovery state. Alert thresholds should reflect the actual RPO and RTO required by each database service.
Failure testing is necessary because a replication topology that appears healthy during normal operation may behave differently during network partitions, storage exhaustion, abrupt server loss, or heavy transaction load. Operators should validate replica catch-up, WAL retention, promotion, client reconnection, old-primary isolation, and reconstruction of failed replicas. Tests should also verify whether the measured data-loss window and recovery duration remain within the intended RPO and RTO.
PostgreSQL streaming replication should therefore be understood as a continuous WAL transport and replay system rather than a complete HA solution by itself. It supplies synchronized standby database state, while production availability depends on surrounding mechanisms for monitoring, quorum or leader decisions, automated failover, service discovery, fencing, and recovery. This distinction provides the foundation for the next topic, where Patroni coordinates PostgreSQL replication nodes into an automated high-availability cluster.
10.03 PostgreSQL Patroni HA Cluster: Auto Failover [w/Code]
Patroni is an open-source high-availability orchestration framework designed to manage PostgreSQL clusters and automate leader election, replication control, and failover. PostgreSQL streaming replication provides synchronized database copies, but replication alone does not decide which server should become primary after a failure. Patroni adds this coordination layer, transforming multiple PostgreSQL instances into an operational HA cluster capable of responding automatically to node and service failures.
A typical Patroni architecture consists of several PostgreSQL nodes running Patroni agents together with a distributed configuration store or distributed consensus system. Patroni continuously observes PostgreSQL health and cluster membership while coordinating leadership through shared state. Instead of allowing each database server to independently decide whether it should become primary, the cluster uses coordinated leader information to maintain a single authoritative writable PostgreSQL instance.
Distributed consensus is essential because failover decisions must remain coordinated even when servers or network paths fail. Patroni commonly integrates with distributed systems such as etcd, Consul, or ZooKeeper, while deployments may also use supported alternatives depending on architecture. The distributed coordination layer stores cluster metadata and leader information. Patroni nodes compete for or renew leadership according to defined rules, allowing the cluster to distinguish the current leader from replicas.
The Patroni leader normally hosts the writable PostgreSQL primary. It periodically renews a leader lock or equivalent leadership state in the distributed configuration store. As long as the node remains healthy and can maintain this leadership information, other Patroni members continue operating as replicas. If the leader becomes unavailable or can no longer renew its leadership state within the configured timing conditions, the remaining members can begin an election and failover process.
PostgreSQL replicas under Patroni normally use streaming replication to receive WAL from the current primary. Patroni therefore does not replace PostgreSQL\'s native replication mechanism; instead, it manages and coordinates it. PostgreSQL remains responsible for WAL generation, transmission, and replay, while Patroni supervises node roles, replication configuration, promotion, demotion, and topology changes. This separation is important for understanding where database durability ends and HA orchestration begins.
Automatic failover begins when the current leader is considered unavailable according to Patroni\'s health and coordination logic. Eligible replicas are evaluated as candidates for promotion. Replication state matters because promoting a severely delayed replica could increase data loss. After a suitable replica obtains leadership, Patroni promotes its PostgreSQL instance from recovery mode into a writable primary and updates cluster state so that other members can recognize and follow the new leader.
Failover timing requires careful configuration. Patroni parameters governing the control loop, leader lock lifetime, retry behavior, and failover decisions determine how rapidly the cluster reacts to failures. Aggressive timing may reduce service interruption but can make the system more sensitive to temporary network instability or overloaded infrastructure. Conservative timing can reduce unnecessary role changes but increases recovery time. These settings should therefore be aligned with the database service\'s Recovery Time Objective (RTO).
Recovery Point Objective (RPO) depends strongly on PostgreSQL replication mode rather than Patroni alone. With asynchronous replication, transactions acknowledged by the failed primary may not yet exist on the promoted replica. Patroni can automate promotion, but it cannot reconstruct WAL that never reached another server. Synchronous replication can reduce this exposure by requiring selected replicas to confirm WAL progress before commits complete, at the cost of additional transaction latency and availability considerations.
Candidate selection is consequently more than simply choosing any running replica. A production HA design should consider replication lag, timeline state, node health, topology, and whether a member is intended to participate in promotion. Patroni provides mechanisms for controlling member behavior so that particular nodes can serve specialized roles. For example, a remote disaster-recovery replica or delayed replica may be valuable for recovery while being inappropriate as an automatic failover candidate.
Split-brain prevention is one of the most important responsibilities of a PostgreSQL HA architecture. If two database nodes simultaneously accept writes as primary, their histories can diverge and automated reconciliation becomes difficult. Patroni uses coordinated leadership to reduce this risk, but reliable operation also depends on sound network design and operational controls. In critical environments, fencing mechanisms may be added to ensure that an isolated former primary cannot continue accepting application writes.
When a failed former primary returns, it cannot automatically assume its previous role because another server may already have accepted new transactions as the current primary. Its database timeline may have diverged from the cluster. Patroni can coordinate reinitialization or use PostgreSQL recovery mechanisms such as pg_rewind where appropriate. The returning node must be synchronized with the current leader before it can safely rejoin the cluster as a replica.
Patroni exposes a REST API that supports health checks, cluster inspection, and administrative operations. Load balancers, proxies, monitoring platforms, or service-discovery mechanisms can use node state to determine which PostgreSQL instance currently serves the primary role and which instances are replicas. This interface helps separate database role management from client routing, allowing infrastructure components to direct write traffic toward the leader and, when appropriate, read traffic toward replicas.
Client routing remains a separate architectural concern because Patroni does not automatically make every application connection follow a newly promoted primary. Production environments commonly place a routing or proxy layer in front of PostgreSQL or integrate service discovery with application connection logic. Components such as HAProxy can use Patroni health information to identify the writable leader. Connection pools must also discard failed sessions and reconnect applications after failover.
Patroni supports controlled administrative operations in addition to unexpected automatic failover. A planned switchover intentionally transfers leadership from a healthy primary to an eligible replica, which is useful during maintenance, operating-system updates, hardware replacement, or infrastructure migration. Unlike emergency failover, switchover can be coordinated while the existing primary remains available, allowing administrators to reduce disruption and verify replica readiness before changing database leadership.
Cluster topology should account for failure domains rather than merely counting database servers. Three nodes located on the same physical host, storage system, power domain, or network switch may not provide meaningful resilience against infrastructure failure. PostgreSQL and Patroni nodes should be distributed according to the failures the system is expected to survive. Coordination services must likewise remain available because loss of distributed consensus can affect the cluster\'s ability to safely perform leadership changes.
Monitoring a Patroni cluster requires observing both PostgreSQL replication health and orchestration state. Important information includes the current leader, member availability, replication lag, WAL positions, timeline changes, leader-lock status, failover history, PostgreSQL process health, storage capacity, connection load, and distributed coordination service health. Monitoring only PostgreSQL can miss orchestration failures, while monitoring only Patroni can overlook database-level degradation that makes a replica unsuitable for promotion.
Operational testing should deliberately reproduce realistic failure scenarios. Tests can terminate PostgreSQL, stop Patroni, isolate a node from the network, interrupt the coordination service, introduce replication lag, or simulate host loss. Engineers should measure detection time, leader election, promotion time, application reconnection, transaction behavior, and replica recovery. The objective is not simply to demonstrate that failover occurs, but to confirm that the entire database service returns within its intended RTO and RPO.
For robot fleet platforms, Patroni can protect centralized operational databases containing mission assignments, robot status, configuration, authorization, maintenance records, and fleet coordination information. If the active database server fails, automatic promotion can restore centralized data services without waiting for manual database intervention. Robots should nevertheless preserve essential local autonomy so that collision avoidance, safety control, and short-term execution do not depend entirely on continuous database availability.
A practical robot deployment may place PostgreSQL and Patroni nodes across independent edge or on-premise servers while keeping low-level robot control outside the HA database path. The database cluster maintains persistent fleet-level information, whereas robots and edge controllers continue safety-critical operations locally. After a database failover, applications reconnect to the new primary and synchronize operational state, limiting the effect of infrastructure failure on physical robot behavior.
Security must also cover the entire HA control plane. PostgreSQL authentication, replication credentials, Patroni API access, coordination-service permissions, network segmentation, encryption, and secret management should be treated as related controls. An attacker capable of manipulating leadership information or administrative APIs could affect database availability even without directly modifying application tables. HA infrastructure therefore requires security controls comparable to those protecting the database itself.
Patroni should ultimately be understood as the orchestration layer surrounding PostgreSQL streaming replication rather than as a replacement for PostgreSQL replication. PostgreSQL maintains replicated data through WAL, while Patroni determines roles, coordinates leadership, automates promotion, and manages cluster membership. Combined with distributed coordination, client routing, monitoring, fencing, backups, and tested recovery procedures, it provides a foundation for automated PostgreSQL high availability in robot and Physical AI platforms.
10.04 MySQL / MariaDB Group Replication [w/Code]
MySQL Group Replication is a distributed replication technology designed to maintain a group of MySQL servers that coordinate database changes and provide fault tolerance. Instead of depending only on a traditional primary-to-replica replication chain, members participate in a replication group and exchange transaction information through a group communication mechanism. This architecture can support automated membership management, failure detection, transaction coordination, and high-availability database services.
Group Replication operates by combining MySQL transaction processing with distributed agreement among participating members. Transactions are first executed locally, but before they are finally committed, information about their writes is distributed to the group for certification. Members evaluate whether concurrently executed transactions conflict with one another. Transactions that successfully pass certification can be applied consistently, while conflicting transactions may be rejected and rolled back.
The certification process is a fundamental difference from simple asynchronous replication. Group members examine transaction write sets representing the rows affected by transactions and use them to identify conflicting updates. This approach enables multiple servers to coordinate transaction ordering without simply replaying an unconstrained stream of independent writes. Applications must nevertheless be prepared for transactions to fail certification and should implement appropriate retry behavior.
MySQL Group Replication can operate in single-primary mode or multi-primary mode. Single-primary mode designates one member as the writable primary while the remaining members normally serve as read-only replicas. This model simplifies application behavior because writes have a clear destination. If the primary leaves the group or fails, another eligible member can be selected, allowing the database service to continue with a new primary.
Multi-primary mode allows multiple members to accept transactions concurrently. This can distribute write activity and remove the requirement that every application write target a single server. However, concurrent modifications to overlapping data can create certification conflicts. Multi-primary deployment therefore requires careful workload design, transaction retry handling, application awareness, and avoidance of write patterns that repeatedly modify the same records across different members.
Group membership is managed dynamically as servers join, leave, fail, or recover. Members exchange information through the group communication layer and maintain a consistent view of participating nodes. Failure detection allows remaining members to update their membership view when a server becomes unreachable. This group-oriented model provides the coordination required for automated HA behavior, but reliable network connectivity and appropriate timeout configuration remain important operational requirements.
Quorum is essential to maintaining safe group operation. A replication group must retain a majority of members to make progress safely when communication failures divide the cluster. For example, a three-member deployment can generally tolerate the loss of one member while retaining a majority. If a network partition isolates nodes without a majority, those nodes should not independently continue normal group decisions, helping reduce the possibility of divergent database histories.
Odd-sized groups are commonly useful because they provide clear majority decisions without adding unnecessary voting ambiguity. Three members are a practical minimum for tolerating one member failure while maintaining majority availability, while larger groups can tolerate additional failures depending on topology. Simply adding members does not automatically improve performance because each member participates in communication, certification, replication, and state maintenance, increasing coordination overhead.
MySQL Group Replication relies on replicated transaction information rather than shared storage. Each participating server maintains its own database copy and local storage subsystem. This eliminates a single shared database disk as the fundamental replication dependency, but consistency also depends on each member correctly receiving and applying replicated transactions. Storage latency, CPU pressure, network delay, and transaction volume can therefore influence replication throughput and member health.
A joining or recovering member must obtain the database state required to become consistent with the group. MySQL can use recovery mechanisms to transfer transactions that the member missed while unavailable, and appropriate provisioning may be required when the difference is too large or necessary history is unavailable. Administrators should monitor recovery progress because a server listed in the topology is not necessarily ready to carry production traffic immediately after starting.
Binary logging remains an important part of the replication architecture. Group Replication depends on transaction metadata and configuration that allow changes to be identified and replicated consistently. Server identifiers, global transaction identifiers, row-based logging, and other replication-related settings must be correctly configured. Inconsistent configuration among members can prevent nodes from joining the group or produce operational behavior different from the intended HA design.
Global Transaction Identifiers (GTIDs) provide unique identities for replicated transactions and help servers track which transactions have already been executed. This is valuable when members leave and later rejoin because replication recovery can determine which transaction history is missing. GTID-based tracking also simplifies many operational tasks compared with relying only on binary-log file names and positions, particularly when database roles change during failover.
MySQL InnoDB Cluster builds a broader HA solution around Group Replication. Group Replication supplies distributed database membership and transaction replication, while MySQL Router can direct application connections toward appropriate cluster members. MySQL Shell provides administration and cluster-management capabilities. Together, these components can create an integrated architecture in which replication, topology management, routing, and administrative automation cooperate rather than being independently assembled.
MySQL Router is especially important when the writable member changes. Applications should not require manually updated database addresses whenever a failover occurs. A routing layer can maintain knowledge of the cluster topology and direct read-write connections toward the appropriate primary while optionally routing read-only traffic toward replicas. Existing application connections may still fail during role changes, so connection pools and applications need reconnect and retry logic.
MariaDB provides related high-availability approaches, but its replication architecture should not be assumed to be identical to MySQL Group Replication. MariaDB deployments commonly use technologies such as Galera Cluster for virtually synchronous multi-primary clustering, alongside traditional replication options. Although both ecosystems address database redundancy and availability, their protocols, configuration, transaction behavior, failure handling, and operational tooling differ and should be evaluated independently.
Galera-based MariaDB clustering replicates transactional changes across cluster nodes while using certification-based mechanisms to coordinate concurrent writes. Multiple nodes can accept application traffic, but this does not mean that conflicting writes become free of coordination cost. Workloads with frequent concurrent updates to the same records can experience certification conflicts or rollbacks, making application transaction design and retry behavior important for stable multi-primary operation.
Network latency has a stronger architectural effect in coordinated replication than in loosely coupled asynchronous replicas. Transactions that require communication or certification across members are affected by communication delay and cluster health. Nodes should therefore be placed according to realistic latency and failure-domain requirements. Spreading replicas across independent infrastructure improves resilience, but excessive geographic distance can directly affect transaction performance and recovery behavior.
Database consistency should be considered together with availability requirements. A robot fleet database may contain mission ownership, configuration, authorization, maintenance state, and scheduling information for which conflicting concurrent updates are undesirable. Single-primary operation may therefore provide a simpler operational model for many transactional workloads. Multi-primary operation can be appropriate when the data model and workload are deliberately designed to minimize conflicting writes.
Robot systems should also separate database HA from real-time robot safety. A MySQL or MariaDB cluster can maintain fleet-level operational state, business records, user information, maintenance history, and mission metadata, but low-level motion control and collision avoidance should not require continuous access to the central database. Edge controllers and robots can preserve essential local operation while the database cluster detects failures, changes membership, and restores centralized service.
Monitoring should cover both database performance and distributed cluster state. Important observations include member role, group membership, transaction throughput, certification conflicts, replication queues, recovery progress, connection errors, storage utilization, CPU load, network latency, and failed members. A cluster can appear operational while one member is falling behind or repeatedly leaving and rejoining, so HA monitoring must evaluate redundancy quality rather than only whether a database endpoint responds.
Failure testing should validate member loss, primary failure, network partition, routing changes, node recovery, and transaction behavior during topology transitions. Engineers should measure failure detection, election or role-change time, application reconnection, transaction rollback, and data consistency after recovery. Testing should also verify whether the remaining group retains quorum and whether isolated nodes behave safely instead of continuing as independent writable database services.
For production robotics platforms, the appropriate architecture depends on workload, consistency requirements, latency, and operational complexity. MySQL Group Replication with single-primary operation provides coordinated redundancy with a relatively clear write model, while multi-primary designs offer greater write distribution at the cost of conflict management. MariaDB clustering offers related alternatives with different implementation characteristics. Both require monitoring, routing, backup, recovery testing, and application-aware retry mechanisms.
Group replication should therefore be understood as more than copying data between database servers. It combines replication with membership management, transaction certification, distributed coordination, and failure handling to maintain a coherent database service across multiple nodes. When integrated with routing, monitoring, appropriate quorum design, backups, and tested recovery procedures, MySQL or MariaDB clustering can provide a resilient database foundation for robot fleet and Physical AI services.
10.05 Redis Sentinel Cluster Auto Failover [w/Code]
Redis Sentinel is a high-availability mechanism for Redis deployments that monitors primary and replica instances, detects failures, coordinates automatic failover, and informs clients about the current primary. It is designed for replicated Redis environments where service continuity is required without introducing Redis Cluster sharding. Sentinel therefore focuses on availability and role management, while Redis replication continues to provide copies of the primary dataset across replica nodes.
A typical Sentinel deployment contains one Redis primary, one or more Redis replicas, and multiple independent Sentinel processes. The primary accepts normal write operations and propagates changes to replicas through Redis replication. Sentinel processes continuously o...
![Robot Database and Storage — Ch.10: Database Replication and HA [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbjp07uvlsesy9gh6elq8.png)



