How Confluent Kafka works: infographic covering the big idea, key terms, end-to-end architecture, the produce and consume paths, the cluster internals, reliability, retention, the Confluent ecosystem, security, an e-commerce order flow, myths versus reality and key takeaways

Download the full-size infographic

What you'll learn: how a record travels through Confluent Kafka, from the producer to the partition log to a consumer group; what actually protects your data (acks, replication, min.insync.replicas, idempotence); where exactly-once stops being true; which Confluent services belong to Confluent Cloud and which to self-managed Confluent Platform; and how to onboard Confluent Cloud on AWS and publish a first message from Spring Boot.

This guide turns two infographics into a full walkthrough: how Confluent Kafka works end to end, and how to onboard Confluent Cloud on AWS and publish a message. Everything in the pictures is also in the text and tables below, so it is searchable and screen-reader friendly. Where an infographic was loose, the article follows the official documentation. For the tuning side, read Kafka Scalability Patterns afterwards.

1. The big idea

Kafka is a distributed, durable event log. Producers append events to named topics, Kafka stores them, and any number of consumers read them independently. Per the Confluent documentation, topics are append-only and events are immutable once written. Three properties explain most of the design:

  • Events are stored in topics (append-only logs). Nothing is updated in place.
  • Consumers read independently, at their own pace. The only per-consumer state is the offset in each partition.
  • Events are not removed because one consumer read them. Unlike a classic queue, a read deletes nothing. Data leaves by retention (age or size) or compaction.

Confluent is the company around Apache Kafka. Confluent Cloud is the fully managed service, built on what Confluent calls the Kora engine. Confluent Platform is software you download and run yourself. Both speak the Apache Kafka protocol, so a standard Kafka client works against either.

2. Why it matters and the key terms

Why Kafka matters What it means in practice
Real-time, scalable data pipeline Events flow as they happen, and you scale by adding partitions and consumers
Decouples producers and consumers The order service does not know who reads its events
Multiple use cases from the same data Inventory, payments, analytics and search can all read the same topic
Durable, fault-tolerant, scalable Data is replicated across brokers and survives broker failures

Quick recap of the key terms:

Term Meaning
Topic A named stream of events
Partition An ordered, append-only log within a topic
Offset The position of a record in a partition
Broker A Kafka server that stores data and serves clients
Consumer group A group of consumers that share a topic's partitions
ISR In-sync replicas: the replicas that are caught up with the leader

The producer writes events and the consumer reads them. For background, see event streaming and CQRS and event sourcing.

3. End-to-end architecture

Event sources (business services, databases through change data capture, IoT devices, logs and files) feed Kafka through producers or Kafka Connect source connectors. Brokers hold the topics. Consumer groups and stream-processing jobs read them, and downstream systems (databases, search, analytics, notifications) receive the results.

Figure 1. End-to-end flow from event sources to downstream systems.

{ "title": "End-to-end Confluent Kafka architecture: sources, producers, Kafka cluster, consumer groups, stream processing and downstream systems", "nodes": [ {"id":"src","label":"Event sources","sub":"services, IoT, logs","glyph":"service","at":[0,0.4]}, {"id":"conn","label":"Kafka Connect","sub":"source connectors","glyph":"mesh","at":[0,1.8]}, {"id":"prod","label":"Producers","sub":"send events","glyph":"worker","at":[1,0.4]}, {"id":"kafka","label":"Kafka cluster","sub":"brokers, topics","logo":"kafka","at":[2,1.1],"tone":"accent"}, {"id":"cg","label":"Consumers","sub":"consumer groups","glyph":"worker","at":[3,0.4],"stack":true}, {"id":"sp","label":"Stream processing","sub":"Streams, Flink","glyph":"stream","at":[3,1.8]}, {"id":"down","label":"Downstream systems","sub":"DBs, search, BI","glyph":"database","at":[4,1.1]} ], "zones": [{"label":"Confluent Kafka (event streaming layer)","nodes":["kafka","cg","sp"]}], "edges": [ {"from":"src","to":"prod"}, {"from":"prod","to":"kafka","label":"produce"}, {"from":"conn","to":"kafka","label":"CDC, files","kind":"async"}, {"from":"kafka","to":"cg","label":"poll"}, {"from":"kafka","to":"sp","label":"poll"}, {"from":"cg","to":"down","kind":"async"}, {"from":"sp","to":"down","kind":"async"} ] } 

How to read this: packets travel along request and response arrows (producers writing, consumers polling). Dashes show event flow into downstream systems.

The infographic's example cluster has three brokers and a topic orders with three partitions and a replication factor of two. That is an illustration: Confluent's introduction notes that a common production setting is a replication factor of 3.

Around the core sits an optional layer of Confluent services (Schema Registry, a monitoring UI, ksqlDB, Cluster Linking, a REST interface). Section 9 says which are Confluent Cloud features and which are Confluent Platform components.

4. The produce path: from app to topic

Step What happens
1. Create event The app builds a record: key, value, timestamp and optional headers
2. Serialize The producer converts key and value to bytes (Avro, Protobuf, JSON or plain strings)
3. Schema (optional) With Schema Registry, the serializer registers or looks up the schema and checks compatibility
4. Get metadata The producer asks a broker which brokers lead which partitions
5. Choose partition The partitioner picks a partition. Records with the same key go to the same partition
6. Batch and compress Records are buffered into batches, optionally compressed (gzip, snappy, lz4 or zstd)
7. Send to leader The batch goes to the leader broker of the target partition
8. Append and replicate The leader appends to the partition log, assigns offsets, and followers replicate it
9. Acknowledge The producer gets an ack according to acks

Figure 2. Sequence of one produced record with acks=all.

sequenceDiagram accTitle: Produce path with acks=all accDescr: The application hands a record to the producer, which serializes it, fetches metadata, batches it and sends it to the partition leader. The leader appends it and a follower replicates it. Once the in-sync replicas have it, the leader acknowledges the producer. participant App as Application participant P as Producer client participant L as Leader broker participant F as Follower broker App->>P: send(topic, key, value) P->>P: serialize, pick partition by key P->>L: metadata request (via bootstrap broker) L-->>P: partition leaders P->>P: batch and compress P->>L: produce batch L->>L: append to partition log F->>L: fetch new records L-->>F: records L-->>P: ack (acks=all, after ISR has the write) P-->>App: send completed

Two details are easy to miss. First, the producer fetches metadata from brokers (any broker in the bootstrap list can answer), not by calling a special controller endpoint. Second, the partitioner only guarantees that equal keys land together. Records without a key are spread across partitions and have no cross-partition order.

The acknowledgment levels, from the Apache Kafka producer documentation:

Setting Meaning Trade-off
acks=0 Do not wait for any acknowledgment. Retries do not take effect Fastest, can lose data silently
acks=1 Wait for the leader only Data can be lost if the leader fails before followers copy it
acks=all Wait for the leader and the in-sync replicas Strongest guarantee, slightly higher latency

In the Apache Kafka 4.1 documentation the default is acks=all and enable.idempotence=true. An idempotent producer gets a producer ID from the broker, and the broker deduplicates retried batches by sequence number, so a retry after a timeout does not write the record twice. Idempotence needs acks=all, retries above zero and max.in.flight.requests.per.connection of 5 or less. Check the defaults of the client version you actually use, because older clients differ.

acks=all plus a suitable min.insync.replicas improves durability, but does not replace monitoring and capacity planning.

5. Inside the cluster: partitions, leaders, followers, ISR and KRaft

A topic is split into one or more partitions. Each partition is an ordered, append-only log, and every record in it has an offset. Order is guaranteed within a partition, not across the whole topic.

Replication. The unit of replication is the topic partition. Each partition has one leader and zero or more followers (a replication factor of 1 has none). Writes go to the leader and followers pull from it.

In-sync replicas. A replica drops out of the ISR if it loses its session or falls too far behind (replica.lag.time.max.ms). Only ISR members can become leader, unless unclean leader election is enabled. It is false by default because it can lose data.

When a write counts as committed. A write is not committed until all in-sync replicas have it, and consumers cannot read it before that. With min.insync.replicas (default 1 at broker level) you set how many replicas, including the leader, must be in sync for a write to succeed.

Replication factor min.insync.replicas acks=all write succeeds while Broker failures tolerated for writes
3 2 at least 2 replicas are in sync 1
3 1 the leader is alive 2, but data can be lost if the leader then fails
2 2 both replicas are in sync 0 (writes stop if either broker fails)

Control plane: KRaft. Modern Kafka manages cluster metadata with a built-in Raft quorum, so no ZooKeeper. A server's process.roles can be broker, controller or both; the Kafka docs recommend 3 or 5 controllers (a majority must be alive) and advise against combined mode in critical deployments. The last bridge release for migrating from ZooKeeper is Kafka 3.9. In Confluent Cloud you do not run controllers, so KRaft matters when you self-manage. The data plane is the brokers: in the infographic, each of the three orders partitions is led by a different broker with a follower elsewhere.

6. The consume path: groups, offsets, rebalancing and replay

Step What happens
1. Subscribe The consumer subscribes to topics with a group.id
2. Group management The group coordinator (a broker) manages membership as consumers join and leave, and assigns partitions
3. Partition rule Within a group, a partition is assigned to at most one consumer at a time
4. Pull and read The consumer polls the partition leaders from its current offsets
5. Process Your code handles the records: update a database, call a service
6. Commit offset Offsets are committed automatically or manually to record progress
7. Replay (optional) A consumer can reread by resetting offsets, as long as the data is still retained

Figure 3. Joining a group, reading and committing.

sequenceDiagram accTitle: Consumer group join, poll and offset commit accDescr: A consumer subscribes with a group id. The group coordinator assigns partitions. The consumer polls the partition leaders, processes records, and commits its offset to the internal offsets topic. If a member leaves, the group rebalances and the new owner resumes from the last committed offset. participant C as Consumer participant G as Group coordinator participant L as Partition leader participant O as Offsets topic C->>G: join group (group.id) G-->>C: assigned partitions C->>L: poll from committed offset L-->>C: records C->>C: process records C->>O: commit offset Note over G,C: a member joins or leaves, so partitions are reassigned G-->>C: new assignment C->>O: read last committed offset

Some facts worth keeping straight:

  • Each partition is consumed by exactly one consumer within each consumer group at any given time. More consumers than partitions means the extras sit idle. Partition count is therefore the ceiling on parallelism for one group.
  • Different groups are independent. Each gets its own logical read of the whole stream.
  • The committed offset is the position of the next record the consumer intends to read, stored in the internal __consumer_offsets topic. If a partition moves to another member, it resumes from the last committed offset, so records after it may be processed again.
  • Rebalancing is triggered by membership or metadata changes. With the classic protocol, pauses can affect every consumer. Kafka 4.0 added a consumer protocol (group.protocol=consumer) that reassigns incrementally, but Confluent's pages differ on its availability, so check your environment. group.instance.id makes a consumer a static member, avoiding rebalances on brief restarts.
  • Defaults in the 4.1 documentation: enable.auto.commit is true, auto.offset.reset is latest, max.poll.interval.ms is 300000.

Replay lets a new service start from the earliest offset, or a fixed bug be reprocessed by resetting offsets. Data that retention has deleted cannot be replayed.

7. Reliability and delivery semantics

Area What Kafka gives you What you still must do
Replication and failover Followers replicate from the leader. If the leader fails, a follower in the ISR becomes leader Choose a replication factor of 3 for important topics, and set min.insync.replicas deliberately
Producer reliability Retries with idempotence avoid duplicate writes Use acks=all, and handle send failures in code
Delivery semantics At-least-once by default Make consumers idempotent, or use transactions where they apply
Operations Monitoring surfaces problems Configure timeouts and retries, a dead-letter topic for poison records, and alert on consumer lag

Delivery semantics, precisely. The Apache Kafka design documentation defines three levels:

  • At most once: messages may be lost but are never redelivered. You get this by disabling producer retries and committing offsets before processing.
  • At least once: messages are never lost but may be redelivered. The documentation states Kafka guarantees this by default. Duplicates appear when a consumer processes a record and then fails before committing.
  • Exactly once: each message is processed once and only once. Kafka supports this within Kafka itself: Kafka Streams, and transactional producers combined with read_committed consumers for Kafka-to-Kafka pipelines.

The real limit of exactly-once. It has scope: the guarantee covers reads, processing and writes that stay inside Kafka. When your code writes to PostgreSQL, calls a payment API or sends an email, Kafka cannot undo or deduplicate that side effect. External systems must cooperate: use idempotent writes (a unique key or upsert) or store the offset with the result in one transaction. Treat exactly-once as effectively once, inside Kafka.

Poison records. One malformed event can block a partition if the consumer retries it forever. Route it to a dead-letter topic after bounded retries. Kafka Scalability Patterns shows that flow.

8. Data lifecycle: retention, compaction and tiered storage

Mechanism What it does Use it for
Retention by time or size Deletes old segments after a time or size limit Operational events, replay windows
Log compaction Keeps at least the latest value for each key State: user profile, product catalogue
Tiered or infinite storage Moves older data to cheaper object storage Long retention without huge broker disks
Replay Consumers reset offsets and reread retained data Reprocessing, new consumers, bug fixes

Retention. Broker defaults: log.retention.hours is 168 (7 days), log.retention.bytes is -1 (no size limit), cleanup policy delete. The infographic's "7 days" and "100 GB" illustrate the two kinds of limit.

Compaction. With cleanup.policy=compact Kafka always retains at least the last known value for each key. Offsets never change and records are never reordered. A key with a null value is a tombstone (a delete marker), itself removed after a retention period. Older values remain readable until compaction runs, so it is a storage policy, not an instant "latest only" view.

Tiered storage. Confluent Platform lists Tiered Storage as a feature that automatically moves data to cost-effective object storage (the infographic mentions S3 and GCS as examples). In Confluent Cloud the documented way to keep data indefinitely is Infinite Storage: setting retention.ms to -1. Use the term for your product.

9. The Confluent ecosystem

The infographic marks these as optional, value-add services. The important detail is where each one lives.

Service What it does Confluent Cloud Confluent Platform (self-managed)
Schema Registry Stores and validates Avro, Protobuf and JSON Schema schemas, and enforces compatibility Yes (part of the Cloud overview, with Stream Governance) Yes
Kafka Connect Ready-made connectors between Kafka and external systems (CDC, files, databases, cloud storage) Yes, as fully managed connectors Yes, you run the workers
ksqlDB SQL over streams (the infographic adds "where supported") Yes, documented as available Yes
Kafka Streams A Java library for stateful stream processing inside your own application It is a client library, so it runs in your app against either Same
Flink Stateful, low-latency stream processing Confluent Cloud for Apache Flink, a fully managed service Confluent Platform for Apache Flink
Control Center Web UI for monitoring and managing Kafka Not listed as a Cloud feature. Cloud has the Confluent Cloud Console, and self-managed Control Center can connect to Cloud Yes
Cluster Linking Mirrors topics from one cluster to another, across regions or environments Yes Yes
REST Proxy HTTP service for interacting with a Kafka cluster Cloud documents a Kafka REST API for producing and administering instead Yes, as REST Proxy
Tiered Storage Moves older data to object storage Cloud uses Infinite Storage Yes

Control Center and REST Proxy are Confluent Platform components. In Confluent Cloud the matching tools are the Console and the Kafka REST API. Schema Registry runs next to Kafka, not inside the broker.

Stream processing choices. Kafka Streams is a library embedded in a Java service, ksqlDB expresses a stream as SQL, and Flink is a separate engine. Pick based on who owns the job and their language. For LLM workloads, see event-driven AI with Kafka Streams.

10. Security and operations

Area What to do
Encryption in transit Use TLS. Confluent Cloud clients must support TLS 1.2
Authentication Confluent Cloud clients use SASL with an API key (PLAIN) or OAUTHBEARER. Self-managed Kafka also supports SASL variants and mutual TLS (mTLS)
Authorization Role-based access control (RBAC) and/or ACLs. Confluent documents RBAC as preferred and says ACLs work together with role bindings
Quotas Limit producer and consumer usage (bandwidth, request rates) so one client cannot starve others
Schema compatibility Define and enforce evolution rules (backward, forward) in Schema Registry
Capacity planning Choose partition count, replication factor and disk for the workload
Key monitoring Under-replicated partitions, ISR changes, request latency, throughput, consumer lag, broker disk and network
Backup and disaster recovery Plan and test recovery: cluster linking or replication, multi-region design, snapshots of configuration

The infographic lists "SASL or mTLS" generically. For Confluent Cloud the client documentation names SASL PLAIN (API key) and OAUTHBEARER. mTLS is a self-managed option.

11. A worked example: e-commerce order flow (illustrative)

This is an educational reference design, not a real company's system. An Order Service publishes an OrderCreated event keyed by orderId to the orders topic, which has three partitions. Three consumer groups read it independently:

  • Inventory group: reserves stock.
  • Payment group: processes the payment.
  • Shipping group: creates the shipment.

Each group publishes the result to an order-status topic, and an analytics sink reads that topic for reporting.

Figure 4. One event, three independent consumer groups.

{ "title": "Illustrative e-commerce order flow: Order Service publishes OrderCreated to the orders topic, three consumer groups react, and results go to an order-status topic and an analytics sink", "nodes": [ {"id":"order","label":"Order Service","sub":"producer","glyph":"service","at":[0,1.2]}, {"id":"orders","label":"orders topic","sub":"3 partitions","logo":"kafka","at":[1,1.2],"tone":"accent"}, {"id":"inv","label":"Inventory group","sub":"reserve stock","glyph":"worker","at":[2,0],"stack":true}, {"id":"pay","label":"Payment group","sub":"process payment","glyph":"payment","at":[2,1.2],"stack":true}, {"id":"ship","label":"Shipping group","sub":"create shipment","glyph":"worker","at":[2,2.4],"stack":true}, {"id":"status","label":"order-status topic","sub":"updated events","logo":"kafka","at":[3,1.2]}, {"id":"bi","label":"Analytics sink","sub":"reporting, BI","glyph":"database","at":[4,1.2]} ], "edges": [ {"from":"order","to":"orders","label":"event","kind":"async"}, {"from":"orders","to":"inv","kind":"async"}, {"from":"orders","to":"pay","kind":"async"}, {"from":"orders","to":"ship","kind":"async"}, {"from":"inv","to":"status","kind":"async"}, {"from":"pay","to":"status","kind":"async"}, {"from":"ship","to":"status","kind":"async"}, {"from":"status","to":"bi","kind":"async"} ] } 

How to read this: dashes show events flowing. Each group reads the whole orders topic independently.

Why it works: keying by orderId keeps one order's events on one partition, so each group sees them in order. A new group (fraud checks, say) can be added without touching the Order Service, and if shipping is slow only its lag grows.

12. Myths versus reality

Myth Reality
Kafka is a queue that deletes on read Kafka is an append-only log. Events stay until retention or compaction removes them
Ordering is global across a topic Ordering is per partition. Use a key to keep related events together
More partitions is always better More partitions allow more parallelism, but add overhead and metadata. Right-size them
Replication is a backup Replication gives availability and fault tolerance. It does not protect against deleted data or bad writes, and it is not a backup
Exactly-once is always guaranteed Exactly-once has scope. It holds within Kafka (transactions, Streams), not for external side effects
Schema Registry is part of Kafka core Schema Registry is a Confluent service that runs alongside Kafka brokers

13. Onboarding Confluent Cloud on AWS: prerequisites and flow

The second infographic covers the practical side: stand up Confluent Cloud on AWS, connect an application privately and publish a message.

Confluent Kafka SaaS on AWS: infographic covering prerequisites, the onboarding flow, the PrivateLink target architecture, the seven steps to publish a message, a Spring Boot producer, the Kafka CLI alternative, verification, security, a production checklist and common pitfalls

Download the full-size infographic

Prerequisites

  • An AWS account, with a VPC and the permissions to create networking resources.
  • A Confluent Cloud account (sign up at confluent.cloud).
  • A network choice. PrivateLink is the infographic's recommendation, with VPC peering as the alternative. Cluster type decides which are available (see below).
  • Client tools: a Java application (Spring Boot) or the Kafka CLI.
  • A basic understanding of topics, API keys and ACLs.

Which cluster types support which network option. The infographic leaves this out. Per the Confluent documentation:

Cluster type Intended use (per Confluent) Private networking on AWS
Basic Development, testing and basic use cases Not described. Public networking
Standard Production workloads that use public networking Public networking
Enterprise Production workloads that require private networking PrivateLink (and Private Network Interface on AWS)
Dedicated High-throughput or private-networking workloads with dedicated capacity PrivateLink, VPC peering and Transit Gateway
Freight Available in select AWS regions Private Network Interface (documented for Freight and Enterprise)

So "PrivateLink or VPC peering" is only a free choice on Dedicated clusters. Enterprise clusters offer PrivateLink but not peering, and Basic and Standard clusters use public endpoints. Options change, so check the cluster types and networking overview pages before choosing.

Overall flow: create the Confluent organization, set up AWS networking, provision the Kafka cluster, configure security (API key and ACL or role), create the topic, publish a message, then verify and monitor.

14. Target architecture: PrivateLink

Figure 5. Applications in your VPC reach a Confluent Cloud cluster over AWS PrivateLink.

{ "title": "Confluent Cloud on AWS with PrivateLink: producer and consumer apps in your VPC reach the Kafka cluster in a Confluent environment through a VPC endpoint", "nodes": [ {"id":"prod","label":"Producer app","sub":"Spring Boot","logo":"spring","at":[0,0.2]}, {"id":"cons","label":"Consumer app","sub":"Spring Boot","logo":"spring","at":[0,1.4]}, {"id":"dns","label":"Private DNS","sub":"Route 53 zone","glyph":"location","at":[1,2.5],"tone":"muted"}, {"id":"vpce","label":"VPC endpoint","sub":"PrivateLink","glyph":"gateway","at":[1,0.8]}, {"id":"cluster","label":"Kafka cluster","sub":"managed by Confluent","logo":"kafka","at":[2.3,0.8],"tone":"accent"} ], "zones": [ {"label":"Your AWS account: VPC","nodes":["prod","cons","dns","vpce"]}, {"label":"Confluent Cloud: environment","nodes":["cluster"]} ], "edges": [ {"from":"prod","to":"vpce","label":"SASL_SSL"}, {"from":"cons","to":"vpce","label":"SASL_SSL"}, {"from":"vpce","to":"cluster","label":"PrivateLink"}, {"from":"dns","to":"vpce","label":"resolves","kind":"replicate"} ] } 

How to read this: client traffic stays on the AWS network from your VPC to Confluent Cloud. Private DNS resolves the cluster's hostname to the endpoint.

What the Confluent PrivateLink documentation requires (this is the Dedicated cluster guide, and Enterprise clusters use a separate guide):

  • A Confluent Cloud network of type PrivateLink for AWS, then a PrivateLink Access that registers your AWS account ID. Wait for its status to move from Pending to Ready.
  • DNS resolution and DNS hostnames enabled on your VPC.
  • A security group allowing inbound TCP on 443 and 9092 (port 80 is optional) from your VPC CIDR. The infographic's "allow egress to VPCe" wording points the wrong way: the rules in the documentation are inbound rules on the endpoint's security group.
  • An interface VPC endpoint built from the Confluent VPC endpoint service name, with a subnet in each Confluent zone, and Enable DNS name unchecked.
  • A private hosted zone in Route 53 for the Confluent DNS domain, associated with your VPC, with a wildcard CNAME for the bootstrap host and one zonal record per availability zone. Broker names in the metadata are not static, so do not hard-code them.
  • Limits: all AWS availability zones are supported except use1-az3 in us-east-1. Cross-region PrivateLink connections are not supported. Missing a zone can cause connectivity problems. The DNS resolution type is fixed when the network is created.

The endpoint cannot be created until the Confluent network and PrivateLink Access exist, so the AWS-side work follows the Confluent-side work.

15. The seven onboarding steps

The infographic shows seven stages in its banner and nine in its detailed list. The detailed list adds "Plan and prepare" (the prerequisites above) and splits "Integrate application" from "Publish message". Here they are as one sequence.

Step Action Detail
1. Organization and environment Sign up, create an organization and an environment (for example prod) Environments group clusters and keep access separate
2. Networking Create the Confluent network and connect it: PrivateLink (recommended), or VPC peering where the cluster type supports it See section 14
3. Cluster Create the Kafka cluster with a type, cloud (AWS), region and, where offered, availability Provisioning takes a few minutes. Match the type to the networking choice
4. Service account and API key Create a service account for the application, grant least privilege, create an API key and secret The secret is shown once, so store it in a secrets manager
5. Topic Create the topic in the Console or CLI Auto-creation of topics is disabled by default in Confluent Cloud
6. Publish Configure the client (bootstrap server, SASL_SSL, API key) and send a message Spring Boot or CLI
7. Verify and operate Consume the message and check it in the Console Then monitor

Step 4 in detail: least privilege. The infographic suggests granting roles such as CloudClusterAdmin, DeveloperWrite (produce) and DeveloperRead (consume). DeveloperWrite and DeveloperRead are real predefined roles, but CloudClusterAdmin is not least privilege: it fully manages one cluster, including ACLs, topics, connectors and ksqlDB. An application that only produces needs nothing like that.

Task Roles in the Confluent documentation
Produce to orders DeveloperWrite on the topic
Consume from orders DeveloperRead on the topic, plus read permission on the consumer group (the docs say a consumer group role is needed to subscribe)
See topics and messages in the Console At least DeveloperRead
Create or delete topics from an app DeveloperManage, which cannot change their configuration

Confluent documents RBAC as preferred over ACLs, and says ACLs work in conjunction with role bindings. Permissions belong to the service account, not to the key. Use one service account per application, avoid user-account keys in production, and rotate keys, which stay valid until rotated or deleted. Find the exact role binding or ACL command for your CLI version in the predefined roles and ACL pages.

Step 5 in detail. The infographic creates orders with 3 partitions and replication 3: three partitions let up to three consumers in one group work in parallel, and 3 is the common production replication factor. Whether the Console form exposes replication depends on the cluster type, so check the topic settings after creation.

16. Publish a message from Spring Boot

Put credentials in environment variables. Never commit an API key or secret.

# application.yml (illustrative values: the hostname is a placeholder) spring: kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS} # e.g. pkc-abc123.eu-west-1.aws.confluent.cloud:9092  security: protocol: SASL_SSL properties: "[sasl.mechanism]": PLAIN "[sasl.jaas.config]": >- org.apache.kafka.common.security.plain.PlainLoginModule required username='${KAFKA_API_KEY}' password='${KAFKA_API_SECRET}'; "[client.dns.lookup]": use_all_dns_ips producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.apache.kafka.common.serialization.StringSerializer acks: all 

The bootstrap, security protocol, serializer and acks properties are listed in the Spring Boot properties appendix. SASL settings have no dedicated Boot property, so they go through spring.kafka.properties.*, which passes raw Kafka client properties (dotted keys in brackets). The client.dns.lookup entry follows Confluent's Java client advice for Cloud, and matters most with PrivateLink where one hostname resolves to several addresses.

import org.springframework.kafka.core.KafkaTemplate; import org.springframework.stereotype.Service; @Service public class OrderProducer { private static final String TOPIC = "orders"; private final KafkaTemplate<String, String> kafkaTemplate; public OrderProducer(KafkaTemplate<String, String> kafkaTemplate) { this.kafkaTemplate = kafkaTemplate; } public void publishOrder(String orderId) { // key = orderId keeps every event for one order on one partition kafkaTemplate.send(TOPIC, orderId, "Order created: " + orderId) .whenComplete((result, ex) -> { if (ex != null) { // log, alert, or route to a retry path: do not ignore a failed send System.err.println("Publish failed for " + orderId + ": " + ex.getMessage()); } else { var meta = result.getRecordMetadata(); System.out.println("Sent to " + meta.topic() + "-" + meta.partition() + "@" + meta.offset()); } }); } } 

KafkaTemplate.send(topic, key, data) returns a CompletableFuture<SendResult<K, V>>; whenComplete handles the outcome without blocking. The infographic's snippet omits the constructor and the result handling, which is where production bugs hide. Use a logger, not System.out, in real code.

The consumer side needs a group and a deserializer. Set spring.kafka.consumer.group-id, auto-offset-reset: earliest for a new group that should read existing data, and the two deserializers:

spring: kafka: consumer: group-id: order-audit auto-offset-reset: earliest key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.apache.kafka.common.serialization.StringDeserializer 
import org.springframework.kafka.annotation.KafkaListener; import org.springframework.stereotype.Component; @Component public class OrderListener { @KafkaListener(topics = "orders", groupId = "order-audit") public void onOrder(String message) { // process idempotently: at-least-once delivery means duplicates can arrive System.out.println("Received: " + message); } } 

The security.protocol, sasl.* and client.dns.lookup settings are common properties, so they apply to the producer, consumer and admin clients at once. For wider Spring Boot practice, see Spring Boot production mistakes.

17. The Kafka CLI alternative

The infographic builds the client config inline with a process substitution. A properties file is easier to read and debug. Keep the file out of source control and out of your shell history.

Source:https://dev.to/avaneeshyadav/how-confluent-kafka-works-architecture-delivery-guarantees-and-publishing-from-spring-boot-on-aws-1afh

Related posts