Amazon Managed Streaming for Apache Kafka (Amazon MSK) now supports in-place upgrades from Standard brokers to Express brokers. With in-place upgrade, you can modernize your existing MSK Provisioned cluster to Express brokers through a fully managed workflow, designed to minimize application changes while helping to maintain high availability throughout. Express brokers deliver increased stability at higher throughputs, scale up to 20 times faster, and recover up to 90% faster than standard Apache Kafka brokers. The in-place upgrade is a direct path to those benefits. Until now, adopting Express brokers meant standing up a new MSK Provisioned cluster and migrating both your data and your applications to it. That cut-over added weeks or months of cross-team coordination and put application availability at risk. The in-place upgrade removes that work entirely. Amazon MSK modernizes your existing cluster to Express brokers through a fully managed workflow, with no new cluster, no application changes, and no downtime, turning a multi-month migration into a few hours.

With the in-place upgrade, Amazon MSK adds Express brokers to your existing cluster and moves topics for you, preserving both the cluster endpoint and your Kafka topic offsets. As partitions move, MSK reassigns their replicas to the Express brokers and transfers partition leadership through Kafka’s standard leader-election process. Producers and consumers continue using the same topics and the same cluster endpoint without interruption. Applications can continue serving traffic throughout the migration, delivering a zero-downtime client experience. Because the upgrade uses Kafka-native protocols that preserve topic offsets, your stateless and stateful applications (including Apache Flink and Kafka Streams) can resume from their exact position, with no reprocessing, errors, or duplicates. The upgrade can be reversed until you decommission the Standard brokers, keeping you in control throughout the process.

Prerequisites

In-place upgrade to Express brokers is available for MSK Provisioned clusters configured with Standard brokers, in all AWS Regions where MSK Express brokers are supported. Before you begin, confirm your cluster meets the following requirements. These are also validated when you assess upgrade readiness with the API, described in the next section:

  • Kafka version: The cluster must run Apache Kafka version 3.9 with KRaft metadata management.
  • Three Availability Zones: The cluster must be deployed across three Availability Zones.
  • No tiered storage support: Clusters with tiered storage enabled are not supported.
  • Topic and broker configuration requirements: Express brokers enforce a set of best-practice configurations to maintain stability and high availability. Before initiating upgrade, confirm your cluster meets the following Express requirements:
    • replication.factor=3.
    • min.insync.replicas=2.
    • unclean.leader.election.enable = false.
    • segment.ms set to at least 60,000 milliseconds.
    • max.message.bytes set to no more than 8 MiB.
    • max.compaction.lag.ms set to at least 24 hours.
    • A supported broker instance type.
    • Per-broker partition counts within the Express broker partition quota.

The upgrade process

The following steps use the AWS Command Line Interface (AWS CLI). You can also perform them programmatically with the AWS Management Console or SDK. To perform the upgrade with an AI agent, use the managing-amazon-msk Agent Skill, which powers agents to manage the upgrade process end to end. All sections of this post can be automated with the managing-amazon-msk Agent Skill.

You can upgrade your existing Standard brokers on M5, M7g or M8g instances to Express brokers on M7g or M8g instances. Express brokers on AWS Graviton4-based M8g instances deliver up to 15% lower compute costs, up to 45% lower P99 produce latency, and 2x faster recovery when consumers fall behind, than Express brokers on M7g. You also no longer pay the broker-side cost when moving data across Availability Zones, which reduces total cross-AZ data transfer costs by 50%.

Pre-upgrade: Assess upgrade readiness

To confirm your cluster is ready for in-place upgrade to Express brokers, use the CreateAssessment API. The assessment validates your broker and topic configurations against Express requirements and returns its findings, including any changes to make first. Address any changes that the assessment flags as blockers before you initiate the upgrade. When you initiate the upgrade, MSK automatically re-validates your configuration and proceeds only after all checks pass.

Create the assessment

To create an assessment, specify the target Express instance type. Express brokers are available in the M7g and M8g instance families. You choose the family, and the upgrade keeps your broker size and broker count. For example, if you are on Standard kafka.m5.large, specify express.m7g.large or express.m8g.large. Clusters with Standard brokers on M8g instances can also upgrade to Express brokers on M8g instances:

aws kafka create-assessment \ --region "$AWS_REGION" \ --cluster-arn "$CLUSTER_ARN" \ --assessment-type UPDATE_BROKER_TYPE \ --assessment-parameters TargetInstanceType=express.m8g.large

Check the assessment status

To check the assessment status, use the following and wait until Status shows COMPLETE (assessments typically finish in about two minutes):

aws kafka describe-assessment \ --region "$AWS_REGION" \ --cluster-arn "$CLUSTER_ARN" \ --assessment-arn "$ASSESSMENT_ARN"

List assessment findings

To read the findings after the assessment is complete, use:

aws kafka list-assessment-findings \ --region "$AWS_REGION" \ --cluster-arn "$CLUSTER_ARN" \ --assessment-arn "$ASSESSMENT_ARN"

ERROR findings are blocking and must be resolved before initiating the upgrade. WARNING and INFO findings are informational. Resolve any blocking findings before continuing.

Performing the upgrade

The in-place upgrade to Express brokers is a managed workflow that you drive at your own pace.

Step 1: Initiate the upgrade

To initiate the upgrade using the CLI, use the UpdateBrokerType API. It takes four inputs:

  • --cluster-arn, the ARN of the cluster you’re upgrading.
  • --current-version, the cluster’s current version (to get it, use the DescribeCluster API).
  • --target-instance-type, a combined string naming your current Standard instance type and the target Express instance type, joined with a hyphen. For example, to upgrade a cluster from kafka.m7g.large Standard brokers to express.m8g.large Express brokers, set the target to kafka.m7g.large-express.m8g.large.
  • --topic-reassignment, how to move topics to Express brokers:
    • AUTO: Amazon MSK manages the topic migration for you. It validates your configuration, provisions Express brokers matching the count and size of your Standard brokers and moves every topic from Standard to Express in safe, managed batches. Your applications keep producing and consuming against the same cluster endpoint throughout. After all topics are on Express and you have validated your applications, you complete the upgrade to decommission the Standard brokers.
    • MANUAL: Amazon MSK provisions the Express brokers but moves no topics until you reassign them yourself, giving you a controlled, topic-by-topic experience. You move and validate topics at your own pace, as described in Step 2.

For example:

aws kafka update-broker-type \ --cluster-arn "$CLUSTER_ARN" \ --current-version "$CURRENT_VERSION" \ --target-instance-type "$TARGET_INSTANCE" \ --topic-reassignment AUTO

Step 2: Move topics (controlled mode only)

When you initiate in controlled mode (--topic-reassignment MANUAL), MSK adds the Express brokers but moves no topics until you drive the reassignments. Wait until the cluster returns to Active with both Standard and Express brokers. Then move topics with the StartTopicReassignment API.

You can selectively move topics with the --topics-to-include and --topics-to-exclude wildcard patterns, and set the direction with --target-instance-type. To move topics to Express brokers, pass your Express instance type ($EXPRESS_INSTANCE, for example express.m8g.large). To move topics back to your Standard brokers, pass your Standard instance type ($STANDARD_INSTANCE, for example kafka.m7g.large). Start with lower-risk topics, validate application performance on Express, then continue.

To complete a migration, all topics (including internal topics) must be moved to the Express brokers. After you finish testing with application topics, confirm you also move internal topics to Express brokers.

Move a specific set of topics onto Express:

For example, to move the orders.* topics to Express brokers:

aws kafka start-topic-reassignment \ --cluster-arn "$CLUSTER_ARN" \ --target-instance-type "$EXPRESS_INSTANCE" \ --topics-to-include 'orders.*'

This moves every topic whose name matches the include pattern onto Express brokers. Patterns are regular expressions, so orders.* can match many topics. It does not name a single topic.

Hold specific topics back from Express:

To move all orders.* topics except the payments topics to Express brokers:

aws kafka start-topic-reassignment \ --cluster-arn "$CLUSTER_ARN" \ --target-instance-type "$EXPRESS_INSTANCE" \ --topics-to-include 'orders.*' \ --topics-to-exclude 'orders\.payments.*'

This moves topics such as orders.created and orders.shipped onto Express, and the orders.payments topics stay on Standard. MSK applies the exclude patterns after the include patterns, so a topic that matches both stays where it is. You can move the held-back topics later in a separate reassignment.

Move all topics at once:

You can also move all topics at once to Express brokers. This is useful if you’ve tested a subset of topics on Express, and are ready to complete moving the remaining topics:

aws kafka start-topic-reassignment \ --cluster-arn "$CLUSTER_ARN" \ --target-instance-type "$EXPRESS_INSTANCE" \ --topics-to-include '.*'

This broad include pattern moves every topic, you can narrow the pattern (or exclude) to move topics incrementally.

Stop in-flight reassignment:

To stop a reassignment, you need its topic reassignment operation ARN. The StartTopicReassignment API returns it in the response. You can also get it from the ListTopicReassignments API, which lists every reassignment on the cluster with its operation ARN and status.

aws kafka list-topic-reassignments --cluster-arn "$CLUSTER_ARN"

Pass the ARN to stop-topic-reassignment:

aws kafka stop-topic-reassignment \ --cluster-arn "$CLUSTER_ARN" \ --topic-reassignment-operation-arn "$OP_ARN"

This stops the in-progress reassignment identified by its operation ARN. Topics that already moved stay on Express brokers. Only one reassignment can run on a cluster at a time, so wait until list-topic-reassignments shows the status as STOPPED. Then start a new reassignment with the same start-topic-reassignment command and patterns you used before. The new reassignment gets its own operation ARN, so use that new ARN if you need to stop it later. To move topics back to Standard instead, set --target-instance-type to $STANDARD_INSTANCE, as shown in the following rollback example.

Roll back process:

To roll back a set of topics by name or wildcard, pass the Standard instance type (for example, kafka.m7g.large) in the --target-instance-type:

aws kafka start-topic-reassignment \ --cluster-arn "$CLUSTER_ARN" \ --target-instance-type "$STANDARD_INSTANCE" \ --topics-to-include 'orders.*'

Targeting the Standard instance type moves the matching topics back to Standard while the rest stay on Express. Move topics again after the issue is resolved and you’re ready to move the topics back to Express. To roll back all topics from Express brokers to Standard brokers, first use the StartTopicReassignment API with .* as the topic wildcard and your Standard instance type as the target instance type:

aws kafka start-topic-reassignment \ --cluster-arn "$CLUSTER_ARN" \ --target-instance-type "$STANDARD_INSTANCE" \ --topics-to-include '.*'

Then, decommission the Express brokers by calling UpdateBrokerType with the Standard instance type as the target instance type:

aws kafka update-broker-type \ --cluster-arn "$CLUSTER_ARN" \ --current-version "$CURRENT_VERSION" \ --target-instance-type "$STANDARD_INSTANCE"

Step 3: Monitor progress

Track the overall upgrade on the Cluster operations tab on the console, or with the DescribeClusterOperation API, passing the operation ARN returned by update-broker-type. In Amazon CloudWatch, graph PartitionCount, BytesInPerSec, and BytesOutPerSec by broker ID to watch partitions and throughput shift from the Standard brokers to the Express brokers.

To check whether the upgrade operation is in progress, complete, or failed, use:

aws kafka describe-cluster-operation --cluster-operation-arn "$OPERATION_ARN"

To show migration progress per topic, so you can see which topics have moved to Express and which are still in flight, use:

aws kafka list-topic-reassignments --cluster-arn "$CLUSTER_ARN"

To list each broker node with its BrokerNodeType, so you can identify which brokers are Standard and which are Express as leadership shifts, use:

aws kafka list-nodes --cluster-arn "$CLUSTER_ARN"

The upgrade takes a few hours depending on cluster size and data volume.

Step 4: Validate applications on Express

After every topic has moved to Express brokers, check your workload before you complete the upgrade. The Standard brokers stay in the cluster during this step, so you can still move topics back to them if you find an issue.

To confirm that all partitions have moved, run list-topic-reassignments. Then check that LeaderCount is 0 on every Standard broker. After that:

  • Verify that topics, partitions, and consumer groups are present.
  • Confirm cluster health: ActiveControllerCount is 1 and OfflinePartitionsCount is 0.
  • Confirm producer and consumer throughput and performance align with pre-upgrade baselines.
  • Confirm stateful applications (for example, Apache Flink and Kafka Streams) remain healthy and continue processing after partition leadership moves to Express brokers.

If a topic has problems, move it back to Standard with the rollback commands in Step 2. Run this check for at least one full processing cycle of your workload before you continue.

Step 5: Complete the upgrade

MSK waits for you to confirm before it removes the Standard brokers. When your validation is done, choose Complete Upgrade on the console, or call UpdateBrokerType again with only the Express instance type:

aws kafka update-broker-type \ --cluster-arn "$CLUSTER_ARN" \ --current-version "$CURRENT_VERSION" \ --target-instance-type "$EXPRESS_INSTANCE"

Amazon MSK decommissions the Standard brokers, and the cluster returns to the ACTIVE state as an Express-only cluster. This step is final: once the Standard brokers are removed, reverting to Standard requires creating a new cluster and migrating to it with Amazon MSK Replicator. After the upgrade completes, update or retire any monitoring alerts and dashboards that are specific to your former Standard brokers.

Step 6: Post-upgrade optimization

After the upgrade, intelligent rebalancing is paused on your cluster. New Express clusters have it turned on by default, but upgraded clusters do not. With intelligent rebalancing, you can scale your clusters up or down by editing the broker count in your clusters in a single action, and Amazon MSK will keep partitions evenly distributed across brokers. Because MSK provisions one Express broker for each Standard broker, your cluster may have more capacity than your workload needs. Turn on intelligent rebalancing first, then use your post-upgrade metrics to right-size the cluster by reducing the broker count or changing the instance size as required.

Additionally, Express brokers surface an improved bootstrap endpoint for client connections. The boot-XXX.clustername.kafka.us-east-1.amazonaws.com endpoints are managed by MSK Express and improve distribution of connections for metadata requests across your brokers. You can update applications to use these new endpoints at any time to improve reliability and connection distribution. Your applications configured with broker ID-based bootstrap endpoints (for example, b-1.clustername.kafka.us-east-1.amazonaws.com) will continue to function after the upgrade, so this step is optional.

Additional considerations

Beyond these prerequisites, plan for the following during the upgrade window:

  • Billing: During the upgrade, MSK adds Express brokers to your existing cluster, matching the size and count of your Standard brokers. For example, for a cluster with 3 kafka.m5.large Standard brokers MSK adds 3 express.m7g.large Express brokers during the upgrade. Because both broker sets are provisioned throughout the upgrade period, you are charged for both broker types until the upgrade completes. There is no additional charge for the upgrade itself.
  • Restricted operations: Some cluster modification operations, such as version upgrades, broker configuration changes, and authentication or monitoring changes, are blocked during the upgrade to prevent configuration mismatches between broker types. Kafka read and write, topic create and delete, and broker reboots continue to work. MSK pauses migration during scaling operations and resumes automatically afterward.
  • Monitoring metrics: Standard and Express brokers emit different CloudWatch metrics. Plan to update dashboards and alarms as topics move to Express. In particular, replace KafkaDataLogsDiskUsed with StorageUsed, and retire Amazon Elastic Block Store (Amazon EBS) storage metrics such as BurstBalance.

Get started with Express brokers on Amazon MSK

The in-place upgrade from Standard brokers to Express brokers gives you a straightforward path to the performance, faster scaling, and lower operational overhead of Express brokers, without application changes, cross-team coordination, or reprocessing state. Amazon MSK handles the broker provisioning, topic movement, and offset preservation for you, whether you let MSK move every topic automatically or move topic by topic. What used to take months of migration effort can now complete in hours.

To move even faster, you can use the managing-amazon-msk skill with your AI coding agent (such as Kiro or any MCP-compatible agent) through the AWS MCP server. The skill assesses your cluster’s compatibility with Express requirements and provides commands to resolve any issues, generates a proof-of-concept package to validate Express performance for your workload, and recommends a right-sized Express cluster footprint.

In-place upgrade to Express brokers is available for MSK Provisioned clusters configured with Standard brokers, in all AWS Regions where MSK Express brokers are supported. To get started, see the Amazon MSK Developer Guide.

 


About the authors

Ankita Mishra

Ankita Mishra

Ankita is a Product Manager for Amazon Managed Streaming for Apache Kafka. She works closely with AWS customers to understand their needs for real-time analytics and high throughput, low latency streaming workloads. Working backwards from their needs, she helps drive the MSK roadmap and deliver new innovations that help AWS customers focus on building novel streaming applications.

Ravindranath Kakarla

Ravindranath Kakarla

Ravindranath is a Senior Software Development Engineer for Amazon Managed Streaming for Apache Kafka. He builds the systems that run customers’ real-time, high-throughput streaming workloads, and recently led the launch of in-place migration from Standard to Express brokers. He also works on cross-region disaster recovery and data distribution, helping customers replicate their streaming data across AWS Regions. Working backwards from customer needs, he focuses on making MSK resilient and simple to operate, so customers can focus on building novel streaming applications.

Ashley Millette

Ashley Millette

Ashley is a Specialist Solutions Architect for Streaming and Analytics at AWS. She partners with customers to design and implement real-time data streaming architectures using services like Amazon MSK helping them build scalable, cost-effective pipelines that turn data in motion into actionable insights. She is passionate about simplifying complex streaming workloads and enabling customers to modernize their data infrastructure with confidence.

Simon Peyer

Simon Peyer

Simon is a Solutions Architect at AWS based in Switzerland. He is a practical doer and passionate about connecting technology and people using AWS Cloud services. A special focus for him is data streaming and automations. Besides work, Simon enjoys his family, the outdoors, and hiking in the mountains.