Atlassian Unveils Data Brewery: A Scalable Synthetic Data Platform for Cloud Migration Testing

Atlassian’s Cloud Transition organization has announced the development and implementation of Data Brewery, a sophisticated synthetic data generation platform designed to facilitate large-scale migrations from on-premises environments to the Cloud. The project marks a significant shift in how the company approaches performance testing and system validation, moving away from traditional API-reliant methods toward a decoupled, two-phase relational generation model. As more enterprise customers prepare for the transition to Atlassian Cloud, the need for high-fidelity, massive-scale datasets has become a critical bottleneck. Data Brewery was built to address this by producing representative data for Jira, Confluence, and various ecosystem applications without the performance overhead associated with standard product interfaces.

The primary objective of the Cloud Transition team is to ensure that migrations are fast, complete, and correct. To achieve this, engineers must be able to stress-test migration tooling against datasets that mirror the complexity and volume of real-world enterprise environments. Early testing efforts revealed that a single data "shape"—defined by the sizes and constraints of various entities—was insufficient. For instance, a system might react differently to tens of millions of Jira work items (bugs, stories, epics, or tasks) concentrated within a few spaces (containers for work) compared to the same volume of data distributed across thousands of spaces. This realization necessitated a platform capable of generating dozens of unique, high-volume datasets at pace.

Initially, Atlassian relied on synthetic data generation tools that functioned through product APIs. While effective for smaller datasets, this model proved untenable for enterprise-level scaling. Product APIs are fundamentally designed for human users or standard integrations; they are not optimized for bulk-generating years’ worth of historical data in a matter of hours. The team encountered several recurring issues, including restrictive rate limits, excessive network overhead, and the constant need to update generation scripts whenever product behavior changed. Furthermore, the sequential nature of API calls made it impossible to achieve the parallelism required for generating millions of records efficiently.

In seeking a more robust solution, the engineering team consulted with senior architects who suggested a pivot in strategy: rather than creating the entire dataset through the product interface, the team should focus on extrapolating a small, valid dataset into a larger one. Research into existing market tools for synthetic data generation also yielded limited results. Most commercial tools utilize a "single-pass model," where each row is created, populated, and linked via foreign keys immediately. While this works for simple flat files or basic relational schemas, it fails under the weight of dense, interdependent enterprise schemas like those found in Jira and Confluence.

Traditional synthetic data generation often breaks down in relational systems for two primary reasons. First, foreign keys in enterprise schemas are rarely simple one-column references. They often involve composite keys—where multiple columns together identify a parent row—or polymorphic foreign keys, where the referenced table changes based on a discriminator field. Second, the requirement that every row be complete upon creation introduces a "scalability tax." Child rows must wait for parent values to be queryable in a database, creating sequencing constraints that limit parallelism and increase database read operations.

The core innovation of Data Brewery is the separation of row generation from relationship resolution. This led to the development of a two-phase generation model. In Phase 1, the platform generates tables independently. Each row is populated with non-foreign-key data, such as IDs, timestamps, names, and text fields, according to a specific configuration. During this phase, foreign-key columns are left intentionally blank or unresolved. The result is a structurally valid but relationally disconnected batch of data. This allows for massive parallelism, as the system can generate a "space" table and a "work item" table simultaneously without one waiting for the other.

Phase 2 involves a dedicated foreign key injector that resolves and links the records. This engine uses valid parent-side records—drawn either from the newly generated Phase 1 data or from an existing database—to populate the empty foreign key fields. Crucially, Phase 2 never creates new rows; it only decides which existing parent each child row should point to. This is where the specific "data shape" is applied, expressed through percentile targets. For example, a configuration might dictate that every space has at least 100 work items (the minimum), while the top 1% of spaces (the p99) exceed 50,000 work items, with a maximum cap of 500,000. By applying these distributions during the injection phase, Data Brewery can mimic the "heavy-tail" distributions found in actual customer data.

The architecture of Data Brewery is defined by a "configuration-first" philosophy. In previous systems, adding support for a new entity required tedious product-specific engineering and API integration. Data Brewery uses a generic domain model where entities, fields, and relationships are defined in YAML or JSON metadata. The platform includes several key abstractions: the Entity (representing business objects like users or comments), the Field (defining data types), the Generator (the logic used to create values, such as sequential IDs or weighted enums), and the Relationship (defining how foreign keys link entities).

By making the system config-driven, Atlassian has reduced the engineering effort required to onboard new products or ecosystem apps. The foreign key injection engine handles complex patterns—including simple, composite, and polymorphic links—through declarative rules. This ensures that the same resolution logic is applied consistently across different products, and complex relational patterns remain visible in the configuration rather than being buried in custom code.

To handle the problem of memory management when dealing with millions of rows, Data Brewery breaks large jobs into smaller, manageable chunks. The system utilizes an "Entity Dependency Graph" to determine the correct order for Phase 2 injection. For a typical dependency chain—Users to Spaces to Work Items to Comments to Attachments—the system first generates all content for all entities in bulk. It then resolves foreign keys for Spaces (linking them to Users), then for Work Items (linking them to Spaces), and so on. This approach removes foreign key resolution from the high-throughput path of row creation, significantly increasing overall performance.

Despite the success of the platform, Atlassian engineers acknowledge ongoing challenges. Moving complexity from code to configuration relocates the problem rather than eliminating it. Two primary areas of focus for the platform’s evolution are managing schema evolution—ensuring that synthetic data remains valid as the underlying Jira or Confluence schemas change—and handling cyclical dependencies, where two entities reference each other.

The development of Data Brewery has provided several generalized lessons for the broader engineering community. The project demonstrated the value of decoupling creation from linkage to unlock parallelism. It also highlighted the benefits of optimizing for the "hot path" by moving expensive operations like relationship resolution to a separate, asynchronous phase. Finally, the shift toward a metadata-driven architecture proved that while implementation complexity may increase initially, the long-term gains in flexibility and developer velocity are substantial.

Today, Data Brewery serves as a foundational tool within Atlassian’s Cloud Transition organization. By generating synthetic datasets that accurately represent the data shapes of the company’s largest enterprise customers, the platform provides the confidence necessary to execute high-stakes migrations. It allows engineers to identify performance bottlenecks and potential data integrity issues long before a customer begins their actual journey to the Cloud, ensuring a more stable and predictable transition for the Atlassian ecosystem.