Atlassian Scales StreamHub to 150 Billion Events per Day Through Architectural Transition to Managed Apache Kafka

Atlassian has successfully completed a multi-year architectural transformation of StreamHub, its centralized event streaming platform, moving from Amazon Kinesis to AWS Managed Streaming for Apache Kafka (MSK). This evolution was necessitated by a massive surge in data volume as the company’s user base expanded and new real-time analytics features were introduced across its product suite. Previously handling a significant 22 billion events per day, StreamHub now ingests and processes approximately 150 billion events daily. The platform delivers over 225 billion events per day, averaging 1.68 million events per second with peak traffic exceeding 3.2 million events per second. This infrastructure powers a wide array of critical functions, including user clicks, impressions, API calls, Change Data Capture (CDC) events, and telemetry.

The decision to migrate away from Amazon Kinesis followed several years of successful operation at a smaller scale. While Kinesis served as an effective "zero-ops" service for volumes up to 30 billion events, the transition to a 5x scale increase exposed four fundamental bottlenecks. First, API reliability was capped at approximately 99.99%, while the team aimed for "five-nines" (99.999%) reliability. Kinesis throttling during traffic spikes frequently impacted producers. Second, the cost of high partitioning became prohibitive. Kinesis scales via shards, each limited to 1 MB/sec ingress and 2 MB/sec egress. Handling Atlassian’s peak loads required thousands of shards, causing monthly costs to grow linearly with traffic. Third, retention limits and cost penalties posed a challenge; storing data beyond 24 hours required expensive add-ons, whereas the team required a seven-day history for safe data replays during downstream failures. Finally, delivery Service Level Objectives (SLOs) were difficult to maintain. StreamHub required local delivery under two seconds and cross-region delivery under three seconds. Meeting these guarantees for high-volume services required Kinesis Enhanced Fan-Out (EFO), which carried a steep price premium.

Apache Kafka was identified as the ideal replacement due to its cost-effective multi-day retention, ability to scale without significant cost increases for additional consumer groups, and support for multi-cloud environments. To minimize operational overhead, Atlassian opted for AWS MSK. This allowed a small platform team to focus on high-level architecture and reliability rather than the granular management of Zookeeper nodes, OS patches, or hardware failures.

A cornerstone of the new architecture is the decoupling of storage and compute through Kafka Tiered Storage. Storing billions of events on high-performance Elastic Block Store (EBS) volumes was deemed too costly. Tiered Storage addresses this by splitting data into two distinct tiers. The Local Tier (Tier 1) utilizes local EBS volumes on brokers to store "hot" data from the last five minutes. This ensures that real-time consumers read from fast, low-latency storage. The Remote Tier (Tier 2) asynchronously offloads closed log segments to Amazon S3, a highly durable and low-cost object store. This tier stores historical data for the seven-day retention period. By implementing this, Atlassian reduced infrastructure costs and prevented heavy historical backfills from starving the Input/Output Operations Per Second (IOPS) required by real-time consumers.

Despite the advantages of managed services, operating at a scale of 150 billion events per day revealed significant technical challenges. The Atlassian team identified several critical areas where architectural assumptions were tested. Broker network limits emerged as a primary constraint; while teams often size clusters based on CPU and memory, network throughput often becomes the bottleneck first. Kafka traffic is amplified by replication, consumer fan-out, cross-region relay, and tiered storage copies. Small broker instances often could not keep up with these secondary workloads during traffic spikes.

Furthermore, the team discovered that Tiered Storage still requires significant local headroom. If a broker is under-provisioned, background workers copying data to S3 compete with active writes and replication. If these workers fall behind, local disks accumulate data intended for S3, leading to linear disk growth and potential exhaustion. S3 request rate limits also became a factor. During a production investigation, the team temporarily increased retention from seven to 21 days and then reverted to seven days. This triggered a "delete storm" of older data that competed with regular writes, slowing down remote copy operations and causing local disk utilization to spike.

Managed service control plane dependencies also introduced risks. During a localized Availability Zone (AZ) degradation, the MSK control plane reported inaccurate status and blocked management actions. This prevented the team from rebooting brokers, scaling the cluster, or increasing disk space at the exact moment those actions were needed. Scaling itself was found to be a slow, asymmetric process. Increasing broker storage is a permanent change that cannot be reversed without creating a new cluster. Broker instance upgrades require a rolling restart taking approximately 15 minutes per broker, making them unsuitable for rapid incident mitigation. Scaling out by adding brokers also failed to provide immediate relief for "hot" topics until partitions were rebalanced, a process that adds further load to an already stressed system.

To harden StreamHub against these vulnerabilities, Atlassian implemented a series of robust solutions. Capacity planning was shifted from aggregate cluster throughput to specific broker-level bottlenecks, including peak ingress, egress, EBS throughput, and remote copy behavior. The team adopted a cultural shift, treating unused broker capacity not as waste, but as a critical reliability feature. Network limits are now intentionally over-provisioned, with clusters designed so that normal peak traffic utilizes only 40-50% of documented broker limits. This provides the necessary "burst" capacity to handle retries and failovers.

Operational safety for Tiered Storage was improved by generating larger segment files and staggering retention changes. Local disks are now sized with emergency buffers to survive S3 offload delays. To protect the Kafka clusters from external shocks, Atlassian built ingress rate limiting and quarantine controls into the StreamHub API. This allows the system to shed or isolate malformed traffic and sudden spikes before they reach the shared broker capacity. Within Kafka, client quotas are used to enforce bandwidth budgets for different identities, preventing a single misconfigured pipeline from monopolizing resources.

To address the limitations of managed control planes, Atlassian implemented a sharding strategy, splitting large clusters into smaller shards to reduce the blast radius of any single failure. They also developed a "failover cluster" runbook. If a primary cluster suffers sustained degradation, the team can provision a clean parallel cluster using Infrastructure as Code (IaC) templates. This process involves four stages: Shard Failover (ETA ~15 mins), Preparation (~2 hours), Evaluation, and finally, Cluster Failover. This provides an "escape hatch" that does not rely on the health of the original cluster’s control plane.

Finally, Atlassian established "companion regions" to ensure compliant regional failover. These are pre-approved zones within the same compliance boundaries that allow producers and consumers to fail over during a regional outage while strictly adhering to data residency requirements.

The transition and subsequent hardening of StreamHub have provided several key lessons for high-throughput streaming platforms. The Atlassian team emphasizes that managed services do not eliminate the need for deep internal expertise, and that reliability is the result of conservative capacity planning, explicit blast radius controls, and tested recovery paths. As Atlassian’s event volumes continue to grow, the company plans to further evolve StreamHub to maintain its resilience and efficiency at an even greater scale.