Live Cross-Region Data Migration Without Perceived Downtime: A Four-Phase Protocol for Multi-Service Cloud Systems

1. Introduction

Modern cloud services must move live data for regulatory data residency, tenant relocation, region expansion, infrastructure retirement, load rebalancing, and disaster-recovery posture while remaining available. The operational difficulty is that the data to be moved is rarely a single table or object. In a messaging platform, a user-visible chat conversation spans metadata, messages, access control lists (ACLs), media attachments, reactions, consumption state, search projections, and routing records owned by different services and stored in heterogeneous backends. Moving the chat conversation therefore requires coordinated relocation of a distributed resource graph, not merely copying rows from one database to another.

The user-facing failure modes are severe. Metadata can reach a target before messages, a deleted message can reappear after cutover, writes can occur after historical copy has passed a record range, and stale routing caches can send requests to both regions or bypass a fence. These scenarios are consistency, privacy, and compliance concerns as well as availability concerns.

The platform studied in this paper serves hundreds of millions of monthly active users. Two production scenarios motivated the work. In the go-local scenario, an entire tenant’s data moves to a region selected for regulatory compliance. In the multi-geo scenario, a single user’s data moves when that user relocates. Tenant and user migrations fan out into one migration workflow per chat conversation; the 6–10 million daily chat-conversation migrations reported later are the aggregate of those workflows. Before the system described here, migration was a manual coordination effort requiring approximately 15 engineers across 5 teams in 2 time zones, working around the clock for 3–5 days of downtime even for small tenants with fewer than one million chat conversations. That process served only a handful of tenants per year. The current system migrates tenants of any size with zero user-visible downtime, no per-migration staffing, and no manual intervention in the common case.

This paper reformulates the goal as zero perceived downtime. Strict zero downtime is difficult to claim in a distributed system that needs a coordination point: at some instant, the system must stop accepting a narrow class of writes long enough to guarantee target completeness and change the routing record. The practical objective is that users do not observe failures, transient restrictions are masked by client retry, and correctness is preserved. The title’s claim of migration without perceived downtime refers to this objective, and Section 6 tests it against the deployed system: no write that met the write lock surfaced to users as an error during the observation window.

The main contributions of this paper are as follows:

1.

Section 4 presents a four-phase migration protocol for resources that span independently owned services, combining cache-convergent pre-locking, selective operation fencing, tombstone-based delete handling, retriable client semantics, and atomic routing cutover, on top of the system model and assumptions defined in Section 3.

2.

Section 4.6 states a cutover-completeness invariant and four safety propositions (no resurrection, no lost write, single write authority, and bounded write fence) with proof sketches under explicit assumptions, including requests served from stale routing caches after cutover.

3.

Section 6 and Section 7 report production telemetry from a 30-day observation window, a trace-driven counterfactual comparison of write availability against stop-and-copy and read-only migration, and a discussion of operational lessons, overhead, applicability, limitations, and threats to validity.

Relative to prior work, the novelty lies in four elements that existing live-migration systems do not provide when one logical resource spans independently owned services: fencing by operation class rather than by phase, so that reads, updates, and deletes are restricted only when and as much as the cutover invariant requires; treating deletion as exportable state during migration, so that a paginated copy cannot resurrect deleted data; enforcing fences through routing metadata that every request already reads, with read-through for resources under migration, so that cutover is safe without a separate lock service; and a single routing transition that makes the target authoritative for all services at once, gated on end-to-end validation through the user-facing application programming interfaces (APIs). Section 2 positions each element against the closest prior work.

The central design observation is that migration availability cannot be solved entirely below the application. Storage systems can copy bytes and replicate changes, and workflow systems can retry activities, but neither layer knows which operations are semantically safe during a region move. In the system considered here, a delete operation changes whether absence should be interpreted as “not yet copied,” “never existed,” or “existed and was intentionally removed.” Similarly, a route change is the externally visible decision that all services agree the target has become authoritative. The protocol therefore separates long migration duration from the short coordination interval that users might perceive.

The remainder of this paper is organized as follows. Section 2 reviews related work and states the novelty of each protocol element against the closest prior mechanism. Section 3 defines the system model, assumptions, and failure model. Section 4 presents the protocol and its correctness arguments, and Section 5 describes the production implementation, including pre-cutover validation. Section 6 evaluates the deployment and compares it with alternative designs. Section 7 discusses lessons, overhead, generalization, and threats to validity, and Section 8 concludes.

2. Background and Related Work

Live migration has been studied in several communities, but the problem addressed here sits at the intersection of storage migration, application-level routing, distributed locking, and workflow orchestration. The system under study is not a general-purpose database and does not own all state in a single storage layer. It is a multi-service application in which each resource service retains authority over its data model and storage implementation. State-of-practice work on microservice data management similarly identifies decentralized ownership and cross-service data consistency as core challenges [1]. This section distinguishes several related lines of work and explains where the present approach differs.

2.1. Database and Storage Migration Services

Managed database-migration products such as AWS Database Migration Service and Google Cloud Database Migration Service move or replicate data between database engines and instances [2,3]. Malhotra et al. evaluate automatic database failover as a way to provide near-zero downtime for cloud-native enterprise applications [4], and Abdel Raouf et al. forecast tenant workload to migrate or replicate multi-tenant databases ahead of demand [5]; both operate within a database tier rather than across independently owned services. Application-level online migrations, such as Stripe’s online migration practice [6], show how to evolve production data while reducing user impact, and cloud-native databases such as Amazon Aurora expose storage-layer designs optimized for high throughput and replication [7]. These systems inform the need to minimize critical sections, but they generally do not solve the case in which one logical user-facing entity spans multiple independently owned services and heterogeneous backends.

Globally distributed databases, including Spanner [8] and CockroachDB [9], provide strong abstractions for replication, transaction ordering, and geo-distributed Structured Query Language (SQL). They can hide many storage-placement changes from the application. However, the platform considered here cannot rely on a single database abstraction. A chat conversation may involve message storage, attachment storage, ACL services, and routing metadata that are not all represented in one database transaction. The migration protocol must therefore coordinate service-level APIs, locks, and routing records above the storage layer.

The key distinction is where the cutover decision is made. In database migration, cutover is often a connection-string change, database role change, or storage-engine metadata transition. In a multi-service application, cutover is a business-level routing decision that must be understood by every service that can serve the resource.

2.2. Live Migration of Virtual Machines and Databases

Live migration of virtual machines (VMs) established the pre-copy pattern: copy most state while the VM runs, then stop briefly to copy final dirty state and switch execution [10]. Post-copy variants invert the ordering, starting execution at the target and fetching missing pages on demand [11]. The analogy is useful because most state should move before the disruptive window, but a chat conversation spans services that can continue accepting semantically different operations.

Database live-migration systems make the analogy more concrete. Albatross migrates tenants in shared-storage databases for elastic cloud environments [12]. Zephyr addresses live migration in shared-nothing databases [13]. Squall provides fine-grained live reconfiguration for partitioned main-memory databases [14]. Other database live-migration systems address latency-aware migration and multi-tenant migration under snapshot isolation [15,16]. These systems show that staged copy and reconfiguration can reduce downtime, but they still assume a database system owns the partitioning and migration mechanism. Our setting moves application resources across service boundaries and must preserve semantics such as deletes, routing, and client retry behavior.

2.3. Online Schema Migration and Zero-Downtime Database Tools

Online schema-migration systems change data shape while reads and writes continue. Google’s F1 schema-change mechanism performs online asynchronous schema changes with careful state transitions [17]. GitHub’s gh-ost, Percona’s pt-online-schema-change, Facebook’s OnlineSchemaChange, MySQL Online Data Definition Language, and Vitess provide mechanisms for online database or schema evolution [18,19,20,21,22].

The present problem differs in two ways. First, the unit of migration is not a table schema or a shard inside one database; it is a logical resource graph across services. Second, the most dangerous operations are application semantic operations. A physical delete is safe in a database sense but unsafe for migration if it erases evidence that a record was deleted. Therefore the design must classify and transform operations, not simply replay database changes.

The transferable principles are a long preparatory phase, a short cutover phase, and the ability to back off when foreground workload is at risk. Our design applies them at a resource-graph level and adds user-level delete semantics.

2.4. Change Data Capture

Change data capture (CDC) systems such as Debezium expose database changes as ordered streams [23]. Change streams can help with migration, but they introduce completion and ordering questions when the logical resource is decomposed across services. The orchestrator must know when every service’s relevant changes have been applied and must coordinate a single cutover point. In our system, the sync-token contract and final write lock provide an explicit completion boundary; a sync token is a service-issued marker used later to request changes after that marker.

Event streams are often optimized for continuous processing, whereas migration needs bounded completeness for a particular resource. The design here avoids requiring a global event-ordering service by moving the boundary into the service export contract: each service must be able to export all relevant records for a sync token and must reject or transform operations that would make that boundary ambiguous.

2.5. Sagas and Durable Workflow Orchestration

Sagas introduced long-running transactions decomposed into subtransactions and compensating actions [24]. Recent work revisits local, global, and saga transaction choices for service-oriented and microservice systems [25]. Modern workflow systems, including Azure Durable Functions, AWS Step Functions, and Temporal, provide durable execution, replay, timers, retries, and checkpointing [26,27,28]. The migration system uses a durable workflow engine as its execution substrate. However, a workflow engine by itself does not define migration semantics: it does not decide which operations to fence, how delete intent is represented, when routing caches are converged, or which invariants must hold at cutover. The contribution here is the migration protocol layered on top of durable orchestration.

2.6. Distributed Locks, Leases, and Fencing Tokens

Distributed coordination systems such as Chubby and ZooKeeper provide lock and coordination primitives [29,30]. Leases provide time-bounded cache consistency and fault-tolerant coordination [31]. Redis Redlock and critiques of distributed locking emphasize fencing tokens and careful failure semantics [32,33]. Kleppmann’s broader discussion highlights dual-write hazards and fencing tokens in the presence of delayed clients [34].

Our system embeds lock state in routing metadata that every request already reads; the authors’ companion paper analyzes this locking mechanism in detail [35]. It uses phased lock escalation, token-gated operations, lease-like expiry, and atomic cutover. This differs from deploying a separate lock service in the request path. The design avoids adding an extra lookup to latency-sensitive operations while still providing service-enforced fences; the cost is that routing-cache convergence becomes part of the lock protocol (Section 4).

2.7. Tombstones, Conflict-Free Replicated Data Types, and Data Residency

Tombstones are common in replicated systems because a deletion must be represented as state long enough to be observed by replicas. Conflict-free replicated data types (CRDTs) formalize related ideas for replicated objects and delete semantics [36]. In our system, tombstones are not used to support arbitrary multi-master conflict resolution; they are used during a bounded migration interval to make delete intent exportable. The tombstone strips user content and personally identifiable information (PII) while retaining the system-generated identifier needed to prevent resurrection.

Data residency and sovereignty concerns provide the operational motivation for many migrations. The General Data Protection Regulation (GDPR) defines data-protection obligations in the European Union [37], and policy analysis has examined cloud computing and national data sovereignty [38]. In production, migration must respect such constraints while preserving availability. The authors’ companion work focuses on contract-based migration when no service knows the full resource graph [39]; the present paper focuses on zero perceived downtime and operation semantics.

2.8. Positioning

Table 1 summarizes the distinction. Prior systems provide building blocks; this work combines them into an application-level protocol for live migration of a distributed resource graph. Because each building block exists separately, the contribution must be judged by what the combination enables that the closest prior mechanism does not. Table 2 states this for each element, and Section 6.5 evaluates what happens when elements are removed: it quantifies fencing all operations instead of only the delta catch-up and blocking deletes instead of tombstoning them, and it explains why omitting the pre-lock is a correctness hazard rather than a performance tradeoff.

3. System Model and Problem Formulation

3.1. Actors

The system has four principal actors. First, clients issue read, write, update, and delete operations against a user-visible resource such as a chat conversation. Clients implement retry logic for retriable service responses. Second, resource services own parts of the logical resource: for example, metadata, messages, ACLs, media references, reactions, and routing metadata. A service is authoritative for its own data model and implements a migration contract. Third, a migration orchestrator coordinates migration through durable workflows. It invokes service APIs, waits for timers, handles retries, and records progress. Fourth, source and target regions host service instances and storage backends. A routing record, held in a globally replicated routing store, maps each logical resource to its active location.

The resource model is intentionally decentralized. The orchestrator sequences operations and coordinates cutover, but each service exports and imports its own records, enforces locks, validates data, and retains authority over service-specific semantics.

3.2. Resource and Operation Model

Let a logical resource G be a set of service-owned records r. Each record is identified by a service, a record identifier, and a location. A record may contain references to other records that are discovered during export. The migration objective is to move the authoritative location of G from source region S to target region T. Table 3 summarizes the notation.

Client operations are classified into four categories:

• Reads: operations that observe state and do not mutate the resource.
• Updates: operations that modify existing records.
• New writes: operations that create new primary records.
• Deletes: operations that remove user-visible records.

The classification is semantic, not merely Hypertext Transfer Protocol (HTTP) method based. A service decides how its API operations map to these categories. The design assumes that services can enforce fences before a restricted operation reaches storage.

The model distinguishes logical authority from physical presence. During migration, records may exist in both source and target. Split brain occurs only if the system allows two authoritative write locations for the same logical operation; the routing record, write lock, and atomic swap prevent that state.

The model also distinguishes user-visible deletion from storage reclamation. A delete is user-visible when the record no longer appears in reads and no longer contains user content. Storage reclamation can occur later. This distinction allows the system to satisfy user semantics immediately while retaining a minimal tombstone needed for migration correctness.

3.3. Goals

The system has four goals.

Availability goal. Reads must remain available throughout migration. Writes should be accepted except during the final synchronization window. During that window, blocked writes must return retriable responses rather than permanent failures.

Consistency goal. At cutover, every record that should be visible at the source must be present at the target, and every record deleted during migration must remain deleted after cutover. The target must not expose partial resource state.

Operational goal. Migration must be automatic, resumable, rate-limited, and observable. Failure should not require manual reconstruction of partially copied state.

Compliance goal. Migration must support region relocation and data-residency requirements while avoiding unnecessary retention of user content.

3.4. Assumptions

The protocol assumes the following:

(A1)

Every resource service implements a migration contract that includes export, import, lock, and location-swap behavior, or participates through an equivalent adapter.

(A2)

The export API supports pagination and a sync token t. A full historical export that issues t returns all records committed before t; a delta export from t returns every record r created, modified, or tombstoned after t was issued.

(A3)

Import is idempotent. Replaying an import page after orchestrator retry does not create duplicate user-visible data.

(A4)

Services can enforce lock states at request handling time, before restricted operations reach storage.

(A5)

Routing metadata is already read on the critical path and can carry migration and lock state without requiring a second database lookup.

(A6)

Clients queue writes rejected with HTTP 423, or equivalent retriable lock responses, and retry them in the background with exponential backoff.

3.5. Failure Model

The design tolerates orchestrator crashes, worker restarts, transient database throttling, network timeouts, partial import failures before cutover, and stale routing-cache entries during a bounded convergence interval. The durable workflow checkpoints after phase boundaries and page-level operations. If a failure occurs before cutover, the source remains authoritative and migration can release locks, clean the target asynchronously, and retry later. A lock that expires before the migration completes fails the migration: it rolls back with the primary location unchanged and is retried automatically with a fresh lock. Transient failures within a phase are retried up to three attempts before the migration rolls back and is rescheduled.

The design does not assume arbitrary Byzantine behavior by services, nor does it solve corruption inside a service that violates the migration contract. It also does not assume that a separate distributed transaction spans all service storage systems. Correctness is achieved by service-local contracts, durable orchestration, sync tokens, tombstones, and atomic routing cutover.

The design makes every phase either repeatable or safely abortable: historical copy is repeatable through idempotent import, write lock is abortable before cutover, cutover is one routing-record transition, and cleanup is asynchronous after user traffic has moved.

4. Design

The design is presented in five parts: the four-phase lifecycle, ending in atomic cutover (Section 4.1); selective operation fencing (Section 4.2); tombstone-based deletes (Section 4.3); retriable client semantics (Section 4.4); and the algorithm with its correctness arguments (Section 4.6). Figure 1 shows the components, Figure 2 the phase timeline, and Figure 3 the delete-handling flow.

4.1. Four-Phase Lifecycle

4.1.1. Phase 1: Pre-Lock

Phase 1 prepares the system for stronger locks without changing user-visible behavior. The orchestrator marks the routing record as migration-in-progress, acquires the pre-lock with a lock token, and waits for cache convergence across routing tiers. Convergence relies on passive expiry of cached routing entries rather than on active invalidation, because invalidation cannot be guaranteed to reach every cache instance across independently operated tiers [35]. A lock token is a unique, expiring value that authorizes the orchestrator to acquire, renew, or release migration locks for G. The wait is set by each resource-owning service’s routing-cache expiry policy; because services configure expiry differently, it typically ranges from 1 to 4 hours across services. For the chat-conversation migrations evaluated in Section 6, the measured Phase 1 duration was 69–98 minutes (Table 6). The convergence delay is the maximum routing-cache time-to-live (TTL) plus replication lag plus a safety buffer; its duration is governed by cache configuration and replication behavior rather than data volume. Each service measures its own maximum cache TTL and declares it in the pre-lock response, the routing store runs with bounded-staleness consistency that caps replication lag at 5 minutes, and the orchestrator adds a safety buffer of approximately 10 minutes (Table 5).

Cache-convergent pre-locking embeds lock intent in routing metadata that request handlers already read. The routing record contains the primary location, secondary location, lock state, token metadata, and expiry. At pre-lock, the orchestrator records T as the secondary location and locks it, so that T does not accept client writes before cutover. After stale cached entries expire, subsequent lock states are visible through the normal routing path.

Separating propagation from enforcement gives every cache tier time to converge before the lock affects user operations.

4.1.2. Phase 2: Hard-Delete Lock and Historical Copy

Phase 2 performs the bulk historical copy while reads and writes continue normally. The only semantic change is that physical deletes are converted to tombstones. The orchestrator exports records page by page from S and imports them into T over Transport Layer Security encrypted channels. Dependency semantics define how a discovered referenced record causes a child workflow for that dependent resource; the companion paper treats this contract in detail [39]. The phase can be long for large resources, but it has no user-facing write fence; the orchestrator extends the hard-delete lock before it expires, for up to two weeks, so that even the largest chat conversations complete historical copy without the lock lapsing (Section 5).

The term hard-delete lock is intentionally narrow. It does not mean that all deletes fail. It means that hard physical deletion is temporarily locked and replaced with a logical tombstone. To the user, the delete succeeds; to the migration system, the delete remains observable.

Historical copy uses pagination to avoid timeouts and excessive memory use. Page boundaries also create recovery points, allowing a restarted worker to resume from a known boundary rather than restarting the whole resource.

4.1.3. Phase 3: Write Lock and Delta Catch-Up

Phase 3 is the only phase that fences user writes. The orchestrator acquires a write lock, preventing new writes, updates, and deletes from reaching storage, and then performs delta catch-up using sync token t from the historical phase. Because most data has already been copied, the remaining delta is designed to be small. Export and import proceed in batches. Before cutover, Phase 3 ends with a validation step that compares source and target (Section 5); validation runs in parallel across time slices of the resource’s lifetime within a strict 5-second budget.

Blocked operations return the retriable HTTP 423 (Locked) response, which clients absorb as described in Section 4.4.

Unlike the hard-delete lock, the write lock is never extended, because every additional second is user-visible. If Phase 3 reaches the write-lock expiry of 5 minutes, which also serves as the abort threshold, the migration fails: the orchestrator releases the lock, allows writes again, cleans the target asynchronously, and retries the migration later with a fresh lock. Because the threshold is the lock expiry itself, a crashed orchestrator also cannot keep writes fenced beyond this bound. Each write-lock expiry is also investigated by operators in parallel with the automatic retry, because an unexpectedly long delta catch-up indicates either a performance problem or an unusual usage pattern. One such pattern is an automated client, such as a misbehaving bot, posting messages to a chat conversation at an extremely high rate, so that the delta never drains; after consulting the bot’s owner, operators block or heavily rate-limit it. This transforms an unexpectedly long critical section into a safe retry and a diagnosable event rather than a prolonged outage.

4.1.4. Phase 4: Atomic Cutover

Phase 4 atomically swaps the primary location and releases the migration locks in a single database operation on the routing record. The previous source location is retained as a secondary location, meaning a non-authoritative routing field used for cleanup and operational diagnosis. Source-region cleanup is scheduled within a 15-day window; this cleanup window is separate from the 30-day storage TTL used for tombstone markers. The atomicity of the routing update is essential: there is no intermediate state in which routing has changed but migration locks remain active, or migration locks have been released while routing still points to the old location. The same write also places a lock on the old location, which is now the secondary location [35]. Stale routing caches do not delay writes after the swap. The shared routing library returns a cached routing record directly only when that record carries no active lock; when it does, the library reads through to the routing store [35]. A record cached during Phase 3 carries the active write lock, so the next request for G reads the post-swap record and is routed to T. The lock on the old location is a backstop: a request that still resolves to S receives the retriable HTTP 423 response instead of committing a write there.

Retaining the previous location is also important for operability. After cutover, operators may need to investigate validation discrepancies, cleanup status, or unexpected service behavior. Keeping it in the routing record makes cleanup a normal follow-on workflow.

4.2. Selective Operation Fencing

The system fences operations by risk, not by phase alone. Table 4 summarizes the service behavior by phase.

This classification preserves most traffic. Reads are always allowed because S remains authoritative until cutover. Updates and new writes are allowed until Phase 3 because the sync-token contract and delta catch-up can capture them. Deletes are transformed during Phase 2 because a physical delete would remove the evidence needed to propagate deletion to T. During Phase 3, deletes are blocked rather than tombstoned because the final delta must have a stable boundary.

Records physically deleted during Phase 1 before historical copy do not reach T; this is correct because the hard-delete lock has not yet defined the migration preservation boundary.

4.3. Tombstone-Based Delete Transformation

Without tombstones, data resurrection can occur. Consider an example message M, a record in G:

1.

Message M is copied from source to target.

2.

A user deletes M at the source while migration continues.

3.

Physical deletion removes M entirely from source storage.

4.

Delta catch-up sees no record for M.

5.

After cutover, M remains in the target and reappears to the user.

Tombstones prevent this sequence by converting the delete into exportable state. The tombstone contains a system-generated record identifier and delete marker, while customer content and PII are stripped immediately. Tombstone markers are cleaned up via storage TTL within 30 days. Figure 3 illustrates the flow.

Absence cannot be exported by a paginated copy, but state can. The tombstone makes deletion a state transition that is visible to the export API while retaining only the minimal system-generated identity required for reconciliation.

The implementation treats tombstone import as idempotent. Reimporting a tombstone leaves the target deleted, and importing a tombstone after a historical live copy overwrites the target with delete intent. The phase ordering and sync-token contract prevent an older live record from superseding a tombstone at cutover.

4.4. Retriable Client Semantics

The write lock would be user-visible if clients treated it as a permanent failure. The protocol therefore depends on retriable semantics. A service returns HTTP 423 for blocked writes during Phase 3; RFC 4918 defines the status code and RFC 9110 defines the underlying HTTP semantics [40,41]. The client queues the rejected write and retries it in the background with exponential backoff, so the user does not see a failure while retries are in progress. The write is delayed for the remainder of the write-lock window, bounded by the values in Table 6. Each client stops retrying after a client-specific threshold, typically a few minutes of exponential backoff; only then is the failure surfaced, and the user is asked to retry manually. Because the write lock is held for 13 s at the median and never beyond its 5-minute expiry, a queued write normally completes well within this retry window. Section 6 measures this outcome on the service side, by tracking whether each write rejected with HTTP 423 eventually succeeds.

This choice moves the boundary between availability and correctness. Instead of accepting writes that could be lost or blocking the entire service, the system rejects a narrow operation class with an explicit, retriable signal. The correctness protocol remains simple, and the user-visible experience remains smooth.

Retriable semantics also prevent ambiguity in service telemetry. A non-retriable error would be counted as a user failure and could cause clients or upstream services to take compensating actions, whereas a retriable lock response communicates that the request was not committed and should be attempted again.

4.5. Algorithm

Algorithm 1 summarizes the service-agnostic lifecycle; service-specific data handling occurs inside export, import, lock, and validation APIs.

Algorithm 1: Four-phase migration for a logical resource

Require:  Logical resource G, source region S, target region T Ensure:  G is authoritative at T, or rollback completes with S authoritative 1: Phase 1: pre-lock and routing-cache convergence 2: Write migration-in-progress state for G into the routing record, record T as a locked secondary location, and acquire the pre-lock with a lock token. 3: Wait for convergence: maximum routing-cache TTL plus replication lag plus safety buffer. 4: Phase 2: hard-delete lock and historical copy 5: Acquire the hard-delete lock for G; services transform physical deletes into tombstones. 6: Start historical export from S and record sync token t. 7: for each export page p for G from S before t do 8:    Import p into T idempotently and checkpoint progress; extend the hard-delete lock if it is about to expire. 9:    Start child workflows for dependent resources discovered in p. 10: end for 11: Phase 3: write lock and delta catch-up 12: Acquire the write lock for G; services return retriable HTTP 423 for blocked mutations. 13: Export from S using t and import the delta into T. 14: Partition the lifetime of G into time slices; in parallel for each slice, read records from S and T through the user-facing APIs, compare record counts, and compare the content of a sample of records. 15: if validation fails or Phase 3 exceeds the 5-minute write-lock expiry then 16:    Release locks, schedule asynchronous target cleanup, and retry the migration later with a fresh lock. return rollback with S authoritative. 17: end if 18: Phase 4: atomic cutover 19: Atomically swap primary and secondary locations in the routing record, clear migration locks, and lock the secondary (old) location. 20: Schedule source-region cleanup within the compliance-accepted cleanup window. return G authoritative at T.

4.6. Correctness Arguments

This subsection states the properties the protocol is designed to maintain and gives a proof sketch for each, under assumptions (A1)–(A6) and the failure model of Section 3. Let ${\tau }_{\mathrm{HD}}$ denote the instant the hard-delete lock is acquired, ${\tau }_{\mathrm{W}}$ the instant the write lock is acquired, and ${\tau }_{\mathrm{C}}$ the instant of the atomic routing swap, with ${\tau }_{\mathrm{HD}}<{\tau }_{\mathrm{W}}<{\tau }_{\mathrm{C}}$ by construction of Algorithm 1.

Invariant (cutover completeness).At ${\tau }_{\mathrm{C}}$, for every record r in G that was present in S at ${\tau }_{\mathrm{HD}}$ or created after ${\tau }_{\mathrm{HD}}$, either r exists in T with the content it has in S, or r was deleted and a corresponding tombstone exists in T.

Proof sketch. The sync token t is issued when historical export starts, after ${\tau }_{\mathrm{HD}}$. By (A2), historical export returns every record committed before t, and the delta export from t, taken after ${\tau }_{\mathrm{W}}$, returns every record created, modified, or tombstoned after t. No record changes in S between ${\tau }_{\mathrm{W}}$ and ${\tau }_{\mathrm{C}}$, because writes, updates, and deletes are rejected before reaching storage (A4). Every record therefore appears in T in its final pre-cutover state, and by (A3) replayed pages do not create duplicates. The swap occurs only after validation succeeds; otherwise the migration rolls back and the invariant is not required to hold at T.

Proposition 1 (no resurrection).A record deleted at S after ${\tau }_{\mathrm{HD}}$ is not readable at T after ${\tau }_{\mathrm{C}}$.

Proof sketch. Between ${\tau }_{\mathrm{HD}}$ and ${\tau }_{\mathrm{W}}$, the hard-delete lock causes the service to write a tombstone instead of removing r (A4). The tombstone is returned by the historical export if it was written before t and by the delta export from t otherwise (A2); in both cases its import replaces any live copy of r at T. Between ${\tau }_{\mathrm{W}}$ and ${\tau }_{\mathrm{C}}$, deletes are rejected, so no delete can bypass the tombstone path. A record physically deleted before ${\tau }_{\mathrm{HD}}$ is not covered by the invariant, and it is also not exported because it no longer exists when historical export runs.

Proposition 2 (no lost write).Every client write acknowledged at S before ${\tau }_{\mathrm{W}}$ is reflected at T at ${\tau }_{\mathrm{C}}$: the record is present with its latest content or, if a user subsequently deleted it, it is absent or tombstoned at T. Every client write rejected between ${\tau }_{\mathrm{W}}$ and ${\tau }_{\mathrm{C}}$ commits only at T when retried.

Proof sketch. A write acknowledged before ${\tau }_{\mathrm{W}}$ was committed at S; it is returned either by historical export or by the delta export from t (A2), so by the invariant the record is present at T in its latest state, which is a tombstone if a user deleted it after ${\tau }_{\mathrm{HD}}$; a record physically deleted before ${\tau }_{\mathrm{HD}}$ is correctly absent. A write attempted between ${\tau }_{\mathrm{W}}$ and ${\tau }_{\mathrm{C}}$ is rejected with HTTP 423 before reaching storage (A4), so it is not acknowledged and is queued and retried by the client (A6). After ${\tau }_{\mathrm{C}}$, a retry that resolves the routing record from cache sees a record that carries the active write lock, so the routing library reads through and routes the retry to T; a request that nevertheless resolves to S is rejected by the lock placed on the old location. The retry therefore commits only at T.

Proposition 3 (single write authority).At no instant are client writes to G accepted at both S and T.

Proof sketch. Client requests are routed to the primary location in the routing record. Before ${\tau }_{\mathrm{C}}$ the primary is S, and the target location is locked so that T does not accept client writes; migration imports write to T during this interval, but they are not client writes, and between ${\tau }_{\mathrm{W}}$ and ${\tau }_{\mathrm{C}}$ no client writes are committed at S. The swap changes the primary to T, releases the migration locks, and locks the old location in one atomic write, so there is no intermediate routing state. After ${\tau }_{\mathrm{C}}$, client writes resolve to T by the argument of Proposition 2, and S rejects them.

Proposition 4 (bounded write fence).No continuous interval during which writes to G are rejected exceeds the write-lock expiry.

Proof sketch. Before ${\tau }_{\mathrm{C}}$, writes are rejected only while the write lock is held (Table 4). The write lock expires after 5 minutes (Table 5), and the orchestrator aborts Phase 3 at the same bound, releasing the lock and re-enabling writes at S before any retry. The bound holds even if the orchestrator crashes, because expiry does not depend on orchestrator progress. After ${\tau }_{\mathrm{C}}$, writes are accepted at T; a request that still resolves to S is rejected by the old-location lock, but the routing library reads through on that request’s locked record, so the next attempt is routed to T and the rejection does not persist.

Liveness follows from the failure model: locks have bounded expiries, durable checkpoints let the orchestrator resume from the last completed page or phase, and a migration that fails rolls back with S authoritative and is retried automatically.

Scope. The invariant and propositions are stated per logical resource G. Dependent resources discovered during export, such as a media object referenced by a message, are migrated by their own child workflows and reach their own cutover independently, so for an interval a migrated chat conversation may reference a resource that is still served from S. Such references remain resolvable because every resource is routed through its own routing record, and the ordering among dependent resources is governed by the dependency semantics of the companion work [39] rather than by the propositions above.

These are proof sketches rather than machine-checked proofs. Each depends on a property that can be tested at a single interface: the invariant and Proposition 1 on the export contract (A2) and tombstone creation; Proposition 2 on service-side fencing (A4) and client retry (A6); Propositions 3 and 4 on the atomic routing write and lock expiry. This decomposition is what allows each service to be validated independently against the contract.

Runtime checking of the invariant. The protocol does not rely on the proof sketches alone. Before release, protocol changes are exercised by continuous fault injection in a staging environment as part of continuous integration. In production, every cutover is gated on the end-to-end validation described in Section 5, which compares record counts and sampled record content of G at S and T while the write lock holds S fixed. A violation of the invariant is visible to this check: a record missing from T, such as a lost write, lowers the target count; a resurrected record, whose tombstone was not exported, remains visible at T but not at S and raises the target count; and divergent content is detected when the record is in the sample. A failed check rolls the migration back before cutover, so a contract violation by a single service becomes a retried migration rather than an incorrect cutover. The check has two limits: content is compared on a sample rather than on every record, and a count comparison cannot detect a missing record and an extra record that cancel exactly. Because the check runs on every migration, it is applied to 6–10 million migrations per day rather than to a test corpus.

The check has detected real contract violations in production. In fewer than 0.00001% of migrations, validation failed and the migration was rolled back before cutover. In each case the cause was a resource-owning service whose export or import did not correctly handle a change in its own record schema or a newly introduced record type, so that records were missing or altered at T. The owning service fixed its implementation, and the migration succeeded on retry. None of these violations reached users, which is the behavior the runtime check is designed to produce: the proof sketches assume that services honor the contract, and validation turns a service that does not into a detected, retried failure.

5. Implementation

The production implementation uses a central migration orchestrator built on a durable workflow engine. The orchestrator runs in a horizontally scalable worker pool and uses persistent checkpoints so that workflow execution can survive worker restarts. It coordinates 15 resource-owning services through a standardized contract. Each service implements service-owned logic for export, import, lock enforcement, and location swap. This division keeps domain knowledge with the owning service and prevents the orchestrator from becoming a monolithic repository of service-specific schemas.

5.1. Service Contract

The core contract consists of four behaviors. Export returns pages of source records and a sync token. Import writes those pages to the target idempotently. Lock acquisition applies a service-specific lock state using a lock token and expiry. Location swap updates routing state, releases the migration locks, and locks the old location, all atomically. Some services also expose dependency-discovery behavior; the companion work treats that contract in more detail [39].

Idempotency is required because durable workflow replay and retry are normal, not exceptional. If an import page succeeds but the acknowledgment is lost, the orchestrator may replay that page. The service must treat repeated imports as convergence on the same target state rather than as duplicate creation.

Export, import, lock, and swap are the minimal behaviors needed to turn a service-owned data fragment into a participant in the global migration protocol.

5.2. Routing-Embedded Lock State

Routing metadata is read on every request to determine the active region. The implementation embeds migration and lock state into that routing record. Request handlers therefore learn the lock state as part of their existing routing resolution. This avoids adding a central lock-service lookup to latency-sensitive paths. The routing library serves cached routing records only for resources without an active lock and reads through to the routing store for resources with one, so the extra read is confined to the small, transient set of resources under migration. The routing record includes the primary location, secondary location, lock details, token metadata, and expiry.

Lock escalation is phased. Pre-lock communicates intent and allows cache convergence. Hard-delete lock changes delete behavior. Write lock fences mutations. The lock token prevents concurrent migrations from acquiring or renewing locks for the same resource, and expiry prevents a crashed orchestrator from holding locks indefinitely. Each lock type has its own expiry, and no lock can outlive the pre-lock. During historical copy, the orchestrator extends the hard-delete lock, together with the pre-lock that bounds it, automatically whenever it is about to expire, so that it does not lapse before historical copy completes; extension is capped at two weeks, the configured maximum, which accommodates the largest chat conversations. The write lock is never extended (Section 4). Table 5 lists the protocol parameters.

Table 5. Protocol parameters used in the production deployment.

Parameter	Value
Pre-lock expiry	48 h plus convergence delay; extended with the hard-delete lock
Hard-delete lock expiry	6 h, auto-extended before expiry up to 2 weeks
Write-lock expiry	5 min, never extended (also the Phase 3 abort threshold)
Phase retry limit	3 attempts for transient failures, then rollback and reschedule
Convergence delay	Service-declared; typically 1–4 h across services
Routing-cache TTL	Per tier; maximum tier TTL 2 h or 4 h depending on service
Resource-owning services	15
Replication-lag bound	5 min (bounded-staleness consistency)
Convergence safety buffer	≈10 min
Client retry	Exponential backoff, typically for a few minutes; client-specific threshold
Source cleanup window	15 days
Tombstone TTL	30 days
Write-lock alert	p99 > 25 s
Pre-cutover validation	Count and sampled-content comparison via user-facing APIs; parallel over time slices; budget 5 s, measured p99 <1 s
Orchestrator worker pool	300+ machines

Table 5. Protocol parameters used in the production deployment.

Parameter	Value
Pre-lock expiry	48 h plus convergence delay; extended with the hard-delete lock
Hard-delete lock expiry	6 h, auto-extended before expiry up to 2 weeks
Write-lock expiry	5 min, never extended (also the Phase 3 abort threshold)
Phase retry limit	3 attempts for transient failures, then rollback and reschedule
Convergence delay	Service-declared; typically 1–4 h across services
Routing-cache TTL	Per tier; maximum tier TTL 2 h or 4 h depending on service
Resource-owning services	15
Replication-lag bound	5 min (bounded-staleness consistency)
Convergence safety buffer	≈10 min
Client retry	Exponential backoff, typically for a few minutes; client-specific threshold
Source cleanup window	15 days
Tombstone TTL	30 days
Write-lock alert	p99 > 25 s
Pre-cutover validation	Count and sampled-content comparison via user-facing APIs; parallel over time slices; budget 5 s, measured p99 <1 s
Orchestrator worker pool	300+ machines

The service request path remains straightforward. It resolves the routing record, checks whether the requested operation is allowed under the observed lock state, and either executes the operation, transforms it, or returns the retriable lock response. The important constraint is that the lock decision occurs before any storage mutation. A service that checks the lock after writing would violate the protocol, even if the check usually succeeds.

5.3. Validation and Rollback

Before cutover, while the write lock is still held, the orchestrator validates the migrated resource end to end. It reads records of G from both S and T through the same user-facing APIs that clients use, rather than through the migration export and import APIs, compares the record counts, and compares the content of a sample of records. Reading through the user-facing APIs means that validation exercises the path users will take after cutover, including service-side transformations and access checks, rather than re-reading the data that the migration contract itself produced. If validation fails, migration uses the rollback behavior defined in Section 3; rollback does not require restoring source state from backup.

Validation adds latency to the write-lock window, which is the only interval users could perceive, so it is engineered to a fixed budget. For a chat conversation, the orchestrator splits the conversation’s lifetime into multiple time slices and validates the slices in parallel, so that validation stays strictly within a 5-second budget regardless of conversation length; in production its p99 duration is under one second. This latency is the price of gating every cutover on an end-to-end check; it is included in the Phase 3 durations reported in Table 6.

Validation is placed before the irreversible routing transition, when a failed validation is still only a failed attempt rather than a user-visible target-state problem.

5.4. Observability

The implementation records per-phase duration, write-lock duration percentiles, first-attempt success, retry success, HTTP 423 rejections per request and whether each eventually succeeded, validation failures, rollback reasons, target-cleanup status, and infrastructure contention signals such as database throttling. Runtime rate limiting can slow migration when background copy competes with foreground traffic.

Observability also serves as a control against silent semantic drift. If a service changes retry behavior, lock enforcement, or delete handling, the aggregate migration success rate may not immediately reveal the issue. Phase-specific metrics, validation failures, and lock-rejection recovery telemetry make such regressions easier to localize.

6. Evaluation

6.1. Methodology

We evaluate the system using production telemetry collected from a structured metrics and logging pipeline over a 30-day observation window. Metrics include write-lock duration percentiles, first-attempt success rate, per-phase duration breakdown, and, for every write request rejected with HTTP 423 during migration, whether that request eventually succeeded after retry. All metrics are collected on the service and orchestrator side from structured telemetry logs aggregated across production regions; client-reported telemetry is not used, and production data is reported only in aggregate form. The telemetry itself is proprietary and cannot be released; to support reimplementation, the protocol is specified completely by the service contract (Section 5), Algorithm 1, the fencing rules in Table 4, and the parameters in Table 5.

The evaluation is observational rather than experimental. Alternative migration strategies were considered during design but were not deployed, because intentionally exposing users to weaker migration strategies would be unacceptable on a production system; Section 6.5 instead bounds their write availability from the measured phase durations.

The observation window captures normal production variation across resource sizes and activity patterns. It is not a synthetic benchmark and does not isolate every independent variable, but it measures the real production system, routing caches, service implementations, and client retry behavior.

6.2. Metric Definitions

Total duration measures elapsed workflow time from pre-lock through cutover, including cache-convergence waiting time.

Phase 1 duration measures pre-lock convergence time, dominated by routing-cache TTLs and replication lag.

Phase 2 duration measures historical copy under hard-delete lock and varies with resource size and dependent-resource discovery.

Phase 3 duration measures the write-lock window during which writes are fenced and retried; it is the primary perceived-downtime metric.

Phase 4 duration measures atomic cutover, which is expected to be below one second because it is a routing-record update and lock release.

First-attempt success counts migrations that complete without requiring migration-level retry. Success after retries counts migrations that eventually complete after automatic retry. Unrecovered lock rejections counts write requests rejected with HTTP 423 during migration that did not eventually succeed after retry. A rejected write that eventually succeeds was delayed but not lost; under the client behavior in (A6), a write is surfaced to the user as a failure only when retries are exhausted, so unrecovered lock rejections are the service-side measure of user-visible errors.

These definitions intentionally separate internal retriable events from user-visible failures. A temporary HTTP 423 during the write lock is not counted as downtime if the request eventually succeeds. Conversely, a small number of unrecovered rejections would matter even if internal workflow success remained high.

6.3. Production Metrics

The system processes 6–10 million chat-conversation migrations per day, with 99.9% first-attempt success, 100% success after retries, and zero unrecovered lock rejections. Over the 30-day window, approximately 1,000 message-send requests were rejected with HTTP 423 because they reached a chat conversation during its write lock. Every one of them succeeded on client retry, and none produced a permanent user-visible error. Because the platform carries more than four billion new messages per day, these rejections affected fewer than one in 100 million message sends, which quantifies how rarely the write fence is encountered at all. Over the same window there were no partial-cutover incidents and no orphaned locks, lock expiry before migration completion occurred in fewer than 0.1% of migrations and was handled by automatic retry with a fresh lock, and no stale-cache write to a source region was detected [35]. Chat conversations vary significantly in size by type: meeting conversations average 10 messages (54% of migrations), group and one-on-one chats average 80 messages (39%), and channel conversations average 4,000 messages (7%, with fewer than 0.1% exceeding one million messages). This distribution explains the Phase 2 long tail: the majority of migrations complete historical copy quickly, while the small fraction of large channel conversations drives the p99.9 and p99.99 outliers.

Table 6. Per-phase duration distribution over 30-day observation window across 6–10 million daily migrations. Phase 1 is governed by routing-cache expiry. Phase 2 and Phase 3 show long tails for large conversations. ^aLower bound: total duration includes Phase 2 for every migration, so its p99.99 is at least the Phase 2 p99.99 (873 min).

Metric	avg	p50	p75	p99	p99.9	p99.99
Total duration	1.2 hrs	1.2 hrs	1.2 hrs	1.4 hrs	2.5 hrs	≥14.5 hrsa
Phase 1 (Pre-Lock)	69 min	71 min	71 min	71 min	74 min	98 min
Phase 2 (Hist. copy)	1.5 min	0.4 min	0.5 min	14 min	130 min	873 min
Phase 3 (Write Lock)	14 s	13 s	15 s	19 s	136 s	159 s
Phase 4 (Cutover)	<1 s	<1 s	<1 s	<1 s	<1 s	<1 s

Table 6. Per-phase duration distribution over 30-day observation window across 6–10 million daily migrations. Phase 1 is governed by routing-cache expiry. Phase 2 and Phase 3 show long tails for large conversations. ^aLower bound: total duration includes Phase 2 for every migration, so its p99.99 is at least the Phase 2 p99.99 (873 min).

Metric	avg	p50	p75	p99	p99.9	p99.99
Total duration	1.2 hrs	1.2 hrs	1.2 hrs	1.4 hrs	2.5 hrs	≥14.5 hrsa
Phase 1 (Pre-Lock)	69 min	71 min	71 min	71 min	74 min	98 min
Phase 2 (Hist. copy)	1.5 min	0.4 min	0.5 min	14 min	130 min	873 min
Phase 3 (Write Lock)	14 s	13 s	15 s	19 s	136 s	159 s
Phase 4 (Cutover)	<1 s	<1 s	<1 s	<1 s	<1 s	<1 s

The 0.1% first-attempt failures arise almost entirely during the write-lock phase from transient database throttling or network timeouts, which in some cases also let the write lock reach its expiry; these succeed on automatic retry. A much smaller class, fewer than 0.00001% of migrations, failed pre-cutover validation because a service did not export or import a changed record schema or new record type correctly; these were rolled back, fixed by the owning service, and succeeded on retry (Section 4.6). Rollback follows the failure model of Section 3.

Figure 4 plots the Phase 2 and Phase 3 percentiles from Table 6: the historical-copy tail grows by more than three orders of magnitude between p50 and p99.99, while the write-lock tail stays within the 5-minute write-lock expiry.

6.4. Comparison With the Previous Manual Process

The system replaced a manual process on the same platform, which provides a measured baseline. Table 7 compares the two using only quantities reported in this paper.

6.5. Trace-Driven Counterfactual Comparison

The alternative migration designs were not deployed, but their write availability can be bounded from the measured phase durations in Table 6. Each alternative performs the same copy work and differs only in when writes are fenced. We assume that each phase would take the same time under each design; this is conservative for the alternatives, because fencing writes earlier does not shorten cache convergence or historical copy. Two properties of the measurements make the comparison exact or one-sided. First, means are additive, so the mean write-blocked window of a design that fences writes during several phases is exactly the sum of the per-phase means. Second, because phase durations are non-negative, every percentile of a sum of phases is at least the same percentile of its largest phase; the percentile values for the alternatives are therefore lower bounds.

We compare four designs: this protocol, which fences writes only in Phase 3; stop-and-copy, which fences writes from the start of historical copy to cutover (Phases 2–3); read-only migration, which fences writes from the pre-lock until cutover (Phases 1–3); and delete blocking, which follows this protocol except that deletes are rejected, rather than tombstoned, during Phases 2–3. Table 8 and Figure 5 report the resulting write-blocked windows.

Relative to stop-and-copy, fencing only the delta catch-up reduces the mean write-blocked window from 104 s to 14 s (7.4×) and the p99.99 window from at least 873 minutes to 159 seconds (at least 329×). Relative to read-only migration, the mean reduction is 303×, because the pre-lock convergence interval, which dominates elapsed time, no longer blocks writes. The tail is where the difference matters for users: under either alternative, at least 1% of migrations would fence writes for 14 minutes or more, over five times the p99.99 write-lock window under this protocol (159 s). Within the protocol’s windows, every rejected message send was recovered by client retry. Fences of 14 minutes or more exceed the few minutes for which clients typically retry (Section 4.4), so queued writes would surface as user-visible errors. Under this protocol, every percentile in Table 8 stays within the 5-minute write-lock expiry.

Delete blocking is the most conservative way to prevent resurrection without tombstones. It confines the fence to deletes, but it rejects them for the same lower-bounded windows as stop-and-copy, so users would see deletes fail for minutes in the tail. Tombstoning removes that window entirely: deletes succeed throughout Phase 2.

Removing the pre-lock is not a downtime alternative but a correctness hazard. Without convergence, a service instance can hold a cached routing record without the lock for up to the maximum routing-cache TTL, which is 2 or 4 hours depending on the service (Table 5); during that interval it could accept writes that bypass the fence and are therefore missing from the delta export. The pre-lock converts this hazard into elapsed time that is not visible to users.

This analysis bounds availability under the measured workload; it does not measure the correctness failures of the alternatives, such as event loss in CDC pipelines or conflicts under fully concurrent copy, which Table 9 summarizes qualitatively.

6.6. Qualitative Comparison of Correctness Tradeoffs

Table 9 complements the counterfactual analysis with correctness and operational tradeoffs that phase durations cannot capture. Scheduled downtime is simple but unavailable. Dual-write, CDC, and fully concurrent approaches reduce downtime but leave partial-failure, completion, or semantic-delete hazards. The presented approach accepts a short retriable write fence in exchange for a clear correctness boundary.

6.7. Operational Observations

Early in production, migrating multiple large chat conversations on the same database physical partition caused partition-level throttling, impacting non-migration traffic. The fix was runtime rate limiting per partition. This lesson shaped the operational posture: migration is background work and must yield to foreground traffic. Write-lock duration is the most closely watched metric. A p99 exceeding 25 seconds triggers immediate investigation.

The evaluation also shows why total duration is not the correct user-impact metric. Most elapsed time is spent in phases that do not block user writes. Phase 1 dominates the median total duration because cache convergence intentionally waits for routing metadata to propagate, and that waiting time is not user-visible. Phase 2 can show long tails for large conversations, but reads and writes continue. Phase 3 is where the system pays the coordination cost needed for correctness; it fences writes, and its p99 remains under 20 seconds in the 30-day observation window.

7. Discussion

7.1. Lessons Learned

Classify operations by risk, not by phase. Phase labels alone are too coarse for live migration. Availability improves when each operation receives the weakest restriction that still preserves the cutover invariant.

Represent deletion as exportable state. Live migration needs a way to distinguish records that were never present from records that were intentionally removed. Encoding delete intent as minimal state preserves that distinction without retaining user content.

Measure write-lock duration at every percentile and alert on regression. Write-lock duration is the most direct proxy for perceived downtime. Average lock time is insufficient because large conversations and high-activity resources create tail risk. Percentile monitoring makes regressions visible before they become user-facing incidents.

Background work must respect foreground traffic. Migration throughput is not the only performance objective. Copying too aggressively can cause shared infrastructure contention that affects unrelated resources. Per-partition throttling and runtime rate limiting are as important as copy parallelism.

Never extend the user-visible lock; investigate it instead. Long-running locks that users cannot see, such as the hard-delete lock, can be extended safely to accommodate the largest resources. The write lock cannot, because every extension is user-visible. Treating a write-lock expiry as a failed, automatically retried migration and as a signal for operator investigation keeps the user impact bounded and surfaces causes, such as a misbehaving high-rate client, that no retry policy would fix.

Make cutover indivisible. Updating location and releasing locks in separate operations creates race windows and cleanup complexity. Keeping primary location, secondary location, and lock metadata in one routing record makes cutover a single state transition.

Validate through the user-facing path, because contracts drift. The correctness violations observed in production did not come from the protocol but from services whose export or import fell behind their own schema and record-type changes. A contract that is correct when a service onboards can become incorrect as the service evolves. Validating through the user-facing APIs before every cutover catches such drift regardless of its cause and converts it into a rolled-back, retried migration.

7.2. Practical Implications

For practitioners, the main implication is that “zero downtime migration” should be decomposed into measurable subproblems: safe operations, transformed operations, fenced operations, and client interpretation of fences. A migration project focused only on copy throughput may still fail users if it does not define delete semantics, routing convergence, and cutover atomicity.

The second implication is organizational. The orchestrator should coordinate, but resource services should own semantics. A central team cannot safely encode every service’s delete behavior, dependency graph, and validation logic. A contract-based approach allows services to participate without surrendering domain authority.

The third implication is that client behavior is part of the migration design. Retrying HTTP 423 converts a necessary write fence into a retriable delay rather than a surfaced failure. Systems that do not control their clients, or whose clients cannot reliably retry, may need a different user-experience strategy such as queueing writes server-side during the lock window. That alternative was not needed in the production system described here, but the design principle remains: the protocol must specify what clients observe, not only what servers do.

Finally, migration should be treated as a product capability rather than an emergency operation. Routine relocation needs stable APIs, automated rollback, telemetry, rollout controls, and clear ownership boundaries so service teams can evolve their storage systems while preserving the migration contract.

7.3. Overhead

The protocol trades user-visible downtime for five bounded costs. First, routing reads for resources under migration bypass the routing cache: the routing library reads through to the routing store only while a resource carries an active lock, so resources that are not migrating incur no extra lookup [35]. Second, tombstones retain a system-generated identifier and delete marker, without user content, for at most the 30-day tombstone TTL. Third, source data remains physically present for up to the 15-day cleanup window after cutover. Fourth, Phase 1 adds 1–4 hours of elapsed time per migration; this time is not user-visible, but it bounds how quickly an urgent relocation can complete. Fifth, pre-cutover validation adds time to the write-lock window, under one second at p99 and never more than its 5-second budget; this is the only overhead that falls inside the interval users could perceive, and time-sliced parallel validation keeps it fixed rather than proportional to resource size. Migration traffic also competes with foreground traffic for shared storage partitions, which the deployment controls through per-partition rate limiting (Section 6).

7.4. Generalization Beyond the Studied Platform

The evaluation covers a single platform, so generality must rest on the assumptions rather than on additional deployments. Two observations bound how narrow the evidence is. First, within the platform the protocol is already applied across heterogeneous participants: 15 resource-owning services with independently chosen storage backends, a set that grew from 7 to 15 without a change to the migration contract [39], and conversation sizes that range from an average of 10 messages for meeting conversations to 4,000 for channel conversations, with a small fraction exceeding one million messages, under the same protocol and parameters. Second, each assumption corresponds to a property that is common outside messaging, as Table 10 summarizes.

Under these assumptions, candidate settings include multi-tenant software-as-a-service applications whose tenant or user data spans several services, such as document collaboration, ticketing, or customer-relationship systems, when they must relocate tenants for residency or capacity. The protocol does not transfer to systems that violate (A5), where there is no per-entity routing decision to swap, or (A6), where clients cannot absorb a retriable rejection; for the latter, Section 7.2 notes server-side queueing as an alternative. These settings are argued from the assumptions and have not been evaluated.

7.5. Applicability and Limitations

Table 10 lists the conditions under which the protocol applies. It is most useful when a logical entity spans multiple storage backends and when downtime is unacceptable but a brief retriable write fence is tolerable; it is less suitable when writes must be accepted continuously with no retriable response. It also requires careful service adoption: a single service that ignores tombstones or violates lock enforcement breaks the resource-level guarantee, and pre-cutover validation converts such a violation into a rolled-back migration only to the extent that the violation changes record counts or sampled content.

The approach also assumes that temporary dual physical residency is acceptable: copied data exists at T before cutover, and source data remains for the cleanup window afterward (Section 7.3). Systems with stricter constraints on temporary duplication would need additional safeguards, such as encrypted staging, shorter cleanup windows, or regulatory approval specific to the migration scenario.

Finally, the migration contract becomes part of the service interface. Changes to export shape, tombstone representation, lock interpretation, or retry semantics must remain compatible with workflows that may already be running.

7.6. Threats to Validity

The evaluation has the following limitations.

External validity. Metrics are from a single messaging platform. Section 7.4 argues generality from assumptions (A1)–(A6) and from the diversity of services and resource sizes within the platform, but other domains have not been evaluated.

Comparison. Alternative strategies were not deployed at comparable production scale. The counterfactual analysis in Section 6.5 bounds their write availability from measured phase durations under the assumption that phase durations would be unchanged; it yields exact means and lower-bound percentiles, while the correctness hazards of the alternatives are compared only analytically. The comparison with the previous manual process (Table 7) is measured on the same platform, but it compares a manual procedure with an automated one rather than two automated designs.

Construct validity. User-visible errors are inferred from service-side telemetry rather than observed on clients. A write rejected with HTTP 423 is counted as recovered if it eventually succeeds; the metric cannot distinguish an automatic background retry from a manual retry by a user who saw an error, so it may undercount user-visible errors, although under (A6) a user sees an error only after the client’s retry threshold is exhausted, and the number of rejected requests is small (approximately 1,000 message sends in 30 days). The rejection count is reported for message sends, the dominant write operation; edits and deletes that met the write lock are covered by the same recovery metric but are not broken out separately. Client-side rendering, latency, and per-client-type coverage are not measured. Phase durations are reported as aggregated percentiles over the window without day-to-day variance or confidence intervals, and telemetry retention is 30 days, so long-term trend analysis is limited to aggregated summaries.

Internal validity. The correctness arguments in Section 4.6 are proof sketches, not machine-checked proofs, and they hold only if every participating service honors the contract. Fault injection is run continuously in a staging environment as part of continuous integration, but not in production, so the production results do not include controlled failure experiments. In production, every cutover is gated on the runtime check of Section 4.6, which is sample-based for content and cannot detect exactly cancelling count errors, but which has detected real contract violations. Production telemetry, including zero unrecovered lock rejections and the 100% eventual success rate, further corroborates the arguments.

An additional threat is that production systems evolve. Cache TTLs, client retry policies, storage throttling behavior, and service dependencies can change. The migration protocol therefore depends on continuous validation and alerting, not only on the original design.

8. Conclusion and Future Work

Zero perceived downtime during large-scale distributed data migration is achievable when migration is designed as a protocol rather than a copy job: operations are fenced by class rather than by phase, deletion is carried as exportable state, fences are enforced through routing metadata that requests already read, and cutover is a single routing transition gated on end-to-end validation. The production system described here replaced a manual migration process requiring 3–5 days of downtime with an automated protocol that migrates 6–10 million chat conversations per day. In the reported 30-day window, write-lock durations stayed under 20 seconds at p99, 99.9% of migrations succeeded on the first attempt and all succeeded after retry, and every message send rejected during a write lock succeeded on client retry. A trace-driven counterfactual analysis shows that fencing writes during historical copy would have raised the tail write-blocked window from 159 seconds to at least 873 minutes, and pre-cutover validation detected and rolled back the rare cases in which a service’s export or import had not kept pace with its own schema changes.

Future work includes formalizing the service migration contract, strengthening automated verification of service implementations, extending pre-cutover validation from sampled to complete content comparison within the same latency budget, and studying how to adapt the protocol to clients with weaker retry behavior. Another direction is to expose richer migration intent to services so that rate limiting can be optimized jointly across resource dependencies and infrastructure partitions.