Atlassian Releases AOSC, an Open-Source Plugin for Zero-Downtime OpenSearch Index Migration and Shard Scaling

Atlassian has announced the release of AOSC, a new open-source engine designed to facilitate zero-downtime index migration and shard scaling for OpenSearch. Developed as a specialized plugin, AOSC (Atlassian OpenSearch Change) addresses a long-standing challenge for database administrators and search engineers: the ability to reshape live indices and scale shards online without interrupting user traffic or risking data inconsistency. The tool is now available to the public via GitHub and official documentation, marking a significant contribution to the OpenSearch ecosystem from Atlassian’s Search Platform team.

The development of AOSC was driven by the internal requirements of Atlassian’s massive search infrastructure. The company’s Core Index Serving team manages the backend for flagship products including Jira, Confluence, and the newly launched Rovo, an AI-powered search and knowledge discovery tool. Atlassian’s search environment is characterized by its immense scale, comprising more than 300 OpenSearch clusters, 2,500 data nodes, and over 1.7 petabytes of data. This infrastructure is distributed across 13 global regions, handling both shared-tenant and per-tenant clusters. As these products evolve and the demand for AI-driven experiences grows, the underlying indices must be frequently modified to improve relevance, isolate large customers, or accommodate expanding datasets.

Historically, reshaping an OpenSearch index—tasks such as changing shard counts, updating mappings, or moving data to new clusters—has been a fraught process. Engineers typically had to choose between several suboptimal methods. The native /_reindex API creates a point-in-time copy, but it fails to capture writes that occur after the process begins, leading to a stale target index. The /_split and /_shrink operations allow for shard count changes but require a total write-block on the source index, resulting in downtime and search staleness. The most robust alternative was "application dual-writes," where the application layer is modified to write to both the old and new indices simultaneously while a backfill is performed. However, Atlassian found that dual-writing often turned a simple administrative task into a multi-week project, introducing significant code complexity and increasing the risk of operational incidents.

To eliminate this "migration tax," the Atlassian team sought a solution that would allow OpenSearch to handle migrations natively and safely on live traffic. The engineering goal was to create a workflow that mimicked online schema changes in the SQL database world. This involves an initial bulk copy of data followed by a Change Data Capture (CDC) pipeline to replay subsequent writes, culminating in an atomic swap of the target and source.

AOSC achieves this by leveraging two specific OpenSearch building blocks: Retention Leases and the Lucene Changes Snapshot. While OpenSearch does not expose an external CDC API, these internal mechanisms allow the plugin to track and replay operations. The migration process begins with the "Backfill" phase, using an Engine.Searcher to pin relevant Lucene segments, ensuring data remains searchable during the scan without preventing ongoing segment merges.

The architecture of AOSC is split into two primary roles: the Coordinator and the Shard Workers. To ensure efficiency and reduce network overhead, AOSC runs its operations inside the cluster, directly adjacent to the shards. The Coordinator manages the overall migration state and records progress within the OpenSearch cluster state—a versioned, cluster-wide metadata system replicated across all nodes. Data nodes listen for updates to this state and advance the Shard Workers accordingly. These workers perform the heavy lifting, reporting periodic progress and synchronization gaps back to the Coordinator via direct transport actions.

The migration workflow follows a strictly orchestrated sequence. First, the migration is initiated and retention leases are acquired for each shard. The backfill process populates the target index while the Lucene changes snapshot begins replaying any writes that occurred during the copy. Once the synchronization gap—the number of operations between the source and target—is reduced to a manageable level (typically fewer than 500 operations), the system prepares for cutover. At this stage, a brief write-block is placed on the source index. The workers perform a final replay of the remaining tail events, the document counts are verified for parity, and an atomic alias swap is performed. Because Atlassian’s Search Platform interacts with clusters through aliases rather than direct index names, the transition is seamless for the end-user.

During the development phase, Atlassian engineers identified several technical hurdles, particularly regarding custom routing keys. In many multi-tenant environments, custom routing is used to ensure related data resides on the same shard. However, Lucene’s delete operations do not naturally carry the original routing key. A naive replay of a delete could lead to the operation being routed to the wrong shard in the new index, leaving the document orphaned. AOSC solves this through sophisticated routing math. When shard counts remain the same, the plugin uses the existing routing. If the shard count is changing (splitting or shrinking), AOSC employs a synthetic-key mechanism and MurmurHash3 routing-shard space logic to ensure deletes are correctly propagated to all potential target shards.

Another critical consideration was the prevention of cluster overload. Early staging tests revealed that an unchecked migration could consume excessive CPU and I/O resources, potentially degrading search performance for other users on a shared cluster. In response, the team designed AOSC to be "deliberately slow and tunable." The plugin includes a comprehensive suite of configuration settings that allow administrators to throttle the migration based on the available headroom of the cluster. Users can limit the number of concurrent shards being migrated, adjust the batch sizes for backfills and replays, and introduce sleep intervals between batches. These "levers" ensure that the background migration remains a steady, low-impact process.

Monitoring was also prioritized to build operator trust. AOSC provides a status API that exposes the real-time phase of every shard, including throughput metrics and the specific replay gap. This data allows engineers to build detailed dashboards to track the convergence of the target index and identify any long-tail shards that may be slowing down the final cutover.

Atlassian’s decision to open-source AOSC was motivated by the realization that index management challenges are universal across the OpenSearch community. By moving the project to the public domain, the team aimed to subject the code to higher discipline, requiring clearer documentation and more robust failure models. The project was also notable for its use of Rovo Dev, Atlassian’s AI coding agent, which assisted in the development process.

The release of AOSC represents a shift in how OpenSearch indices are managed at scale. By moving the coordination problem of index migration from the application layer into the OpenSearch cluster itself, AOSC provides a controlled, repeatable workflow for what was previously a high-risk manual task. The plugin effectively bridges the gap between simple data copying and true online schema evolution.

For organizations looking to implement the tool, Atlassian provides a local "quickstart" guide involving Docker and a set of sample API commands to demonstrate a migration from a source index to a target index with a different shard count. The company emphasizes that while AOSC is not "magic," it provides the necessary infrastructure to perform backfills, replays, and validations in a way that preserves the integrity of live search services.

As OpenSearch continues to grow as a preferred choice for large-scale search and AI applications, tools like AOSC are expected to play a vital role in reducing operational overhead. The project remains hosted under Atlassian Labs, with the team encouraging community contributions and feedback to further refine the engine’s capabilities. With the ability to scale shards and reshape indices without downtime, AOSC offers a path toward fully automated, policy-driven shard scaling, where indices can grow or shrink dynamically in response to data volume without human intervention.