Submitted:
20 October 2026
Posted:
21 October 2026
You are already at the latest version
Abstract
Live cross-region data migration in Internet-scale cloud services needs distributed locking, but a dedicated lock service adds infrastructure, a lookup on the request path, and an atomicity gap between lock release and location cutover. We instead embed lock metadata in the routing records that services already read on every request. The primitives—fencing tokens, leases, and conditional writes—are standard; what changes is where lock state lives. In a messaging platform performing 6–10 million migrations per day, lock checks ride the existing routing lookup, adding a point read only for resources holding an active lock; location swap and lock release become one atomic write; and locks propagate by passive cache expiry. Write-lock duration, the only user-impacting phase, stays under 20 seconds at p99. We compare the design with established lock services and separate measured from inferred benefits.
Keywords:
distributed locking
; live data migration
; routing metadata
; cache convergence
; cloud computing
; data sovereignty
; coordination protocols
1. Introduction
Data sovereignty regulations increasingly require cloud services to move user data across regions while the service stays available. In a messaging platform serving hundreds of millions of users, every live migration needs coordinated locking: concurrent migrations must be prevented, deletes intercepted during bulk copy, and writes briefly blocked before cutover.
The conventional answer is a lock service such as Chubby [1] or ZooKeeper [2]. For live migration at Internet scale it has three drawbacks: it is additional infrastructure to deploy, monitor, and staff; it puts a lock lookup on the request path, because a service cannot know whether a resource is migrating without asking; and it leaves an atomicity gap, because switching location and releasing locks are separate actions, so a crash between them leaves a partial cutover.
Every resource-owning service in our platform already resolves a per-resource routing document on every request. We extended that document to carry lock state, so checking a lock needs no extra lookup unless a lock is active, cutover and release are one atomic write, and correctness rests on token and version checks rather than synchronized clocks. The mechanism is the production locking substrate for residency-driven migration.
We do not propose a new coordination primitive; fencing tokens, leases, and conditional writes are well established. The contribution is architectural—where lock state lives—and its consequences at production scale: (1) a routing-embedded lock design in which lock and routing metadata share one document; (2) a lock lifecycle of token-gated operations, bounded leases, and passive cache convergence, with its timing rationale; and (3) an evaluation that separates measured costs from the inferred comparison with a lock-service baseline, with a failure-mode analysis.
2. System Model and Background
2.1. Platform and Routing Model
The platform consists of many independently owned microservices, each with its own storage, so a single entity, such as a conversation, spans several services.
To route each user request to the region that currently holds an entity’s data, every resource-owning service maintains a routing database: one location document per resource, keyed by resource identifier, holding the entity’s geographic location. This document is backed by Azure Cosmos DB, globally replicated across 12 or more regions, and is resolved on every user request through a shared routing library that each service links at build time. Ordinary point reads of this document are served from a local-region replica; Cosmos DB’s published SLA guarantees read latency under 10 ms at the 99th percentile and typically 4 ms at the median. [3] This per-request routing lookup is the infrastructure our design reuses.
Because every operation resolves the location document before reaching any of the resource’s collections, a lock embedded in that one document applies uniformly to the whole resource, however many stores its data occupies.
2.2. Migration Process and the Need for Locking
Data-residency regulation requires moving an entity’s data between regions while it stays live. Migration runs four ordered phases, whose design, correctness, and orchestration are detailed in companion work: [4,5] a pre-lock records intent and waits for caches to converge (Section 2.4); a hard-delete lock turns physical deletes into tombstones during the bulk copy; a brief write lock freezes writes while the remaining delta is copied; and an atomic cutover switches the location and releases all locks together.
2.3. Consistency Requirements
Correctness for migration means three things. Mutual exclusion: at most one migration may hold and advance the lock for a given entity. No lost updates: no write or delete may slip through to the source region once the corresponding lock phase is in force across the fleet. Atomic cutover: the routing location and the lock state must change together—there must be no observable state in which the location has switched but locks remain, or vice versa. Reads are never blocked in any phase; only writes and deletes are affected, and only briefly.
2.4. Caching and Replication Model
Because the location document is read on every request, services cache it in multiple tiers—for example an edge cache, a regional cache, and a service-local cache—each with its own time-to-live (TTL). TTLs differ by service: one service in our platform uses a maximum tier TTL of 2 hours, another 4 hours. The routing database has a single write region, with read replicas elsewhere, and is configured for bounded-staleness consistency, so a replica’s lag behind the write region has an upper bound enforced by the database. [3] Each resource-owning service configures that bound, trading data freshness against cost; our chat location database uses 5 minutes (Section 4.1). These two quantities—the maximum cache TTL and the bounded replication lag—are what the convergence phase depends on (Section 4).
2.5. Failure and Clock Assumptions
We assume a standard crash-recovery model: services and the orchestrator may crash at any time, messages may be lost, and network partitions may occur. Such failures are routine here; the orchestrator is a durable workflow running across 300 or more continuously recycled hosts. [5]
Crucially, the design makes no assumption of a synchronized global clock. Leases are coarse (hours for the copy phases, minutes for the write phase), so clock drift of seconds is immaterial; correctness comes from token validation and atomic conditional writes (§3), not from comparing timestamps across machines.
2.6. Store-Level Movement vs. Cross-Service Coordination
Managed stores replicate and move data within one storage system—our platform relies on Cosmos DB multi-region replication, [6] and Nomad [7] and Albatross [8] migrate data the same way. But an entity’s data spans many independently owned stores, and no single store owns all of it. Each service moves its own bytes; [4,5] what must be coordinated across services is the locking and cutover, which is this paper’s problem.
3. Design: Routing-Embedded Locks
3.1. Location Document Schema
The location document carries lock metadata alongside its location metadata (Figure 1):
This co-location is the core design decision: (a) a routing lookup returns lock state with no extra call; (b) location swap and lock release are a single atomic write; and (c) every lock mutation is guarded by the document’s ETag. An ETag is an opaque version identifier that the database regenerates on every successful write to a document. A writer supplies the ETag it last read as an if-match condition; the write commits only if the current ETag still matches, and is otherwise rejected atomically (HTTP 412) [9]. Because all writes execute in the single write region (Section 2.4), the check runs against the authoritative copy, never a stale replica: an ETag-conditional write is a compare-and-swap on the whole document, versioned by the store, not by any client clock.
3.2. Lock Enforcement on the Routing Path
For each request, the shared routing library returns the lock state with the location, and the service allows or denies the operation—blocking writes during a write lock, tombstoning deletes during a hard-delete lock. Enforcement is a by-product of the routing lookup.
Records are cached, but a cached record carrying an active, started lock is re-read from the write region, never a read replica, because the orchestrator may have changed the lock since caching (Listing 1). That read-through is the entire residual overhead, confined to the small, transient set of resources with an in-flight lock.
| Listing 1. Read path (resolveLocation) and write path (mutateLock) of the routing library. The read path’s timestamp test (skew: a few seconds) decides only when to read through. Safety is the write path: every acquire, escalation, renewal, and swap is a token check, refused with HTTP 423 (Locked), plus one replace conditioned on the store-assigned ETag. A stale read yields an outdated ETag, so the replace fails (412) and is retried on fresh state. The clock decides only when an expired lease may be taken over; the old token then fails the equality check. |
|
resolveLocation(resourceId): doc = cache.get(resourceId) # cache miss if doc is null: doc = routingDB.pointRead(resourceId) cache.put(resourceId, doc) return doc # cache hit, but lock may have changed if lockActiveAndMutable(doc): # read the write region, not a replica doc = routingDB.pointRead(resourceId, region = WRITE) cache.put(resourceId, doc) return doc lockActiveAndMutable(doc): # no live lock if doc.lockInfo is null or doc.lockInfo.expiry < now(): return false # lock has started; may be mutating if doc.lockInfo.startTime - skew < now(): return true return false mutateLock(resourceId, token, change): loop: doc = routingDB.pointRead(resourceId) live = doc.lockInfo is not null and doc.lockInfo.expiry > now() # acquire: no live lock may exist if token is null and live: return 423 # escalate/renew/swap: caller must hold it if token is not null and (not live or doc.lockInfo.token != token): return 423 newDoc = change(doc) # new token on acquire # compare-and-swap on the store’s ETag if routingDB.replace(newDoc, ifMatch = doc.etag): return newDoc # HTTP 412: doc changed; re-read, re-check |
3.3. Token-Gated Operations
Pre-lock issues a lock token stored in the location document, and every later operation—escalation, renewal, and the final swap—must present it. While a lock is live, the library rejects any mutation without the matching token before it reaches the database (Listing 1), so only the orchestrator that began a migration can advance or complete it. This is Kleppmann’s fencing-token pattern, [10] except that the check lives in the shared library that is the only path to the routing database, so it cannot be omitted.
3.4. Two-Layer Enforcement
Enforcement splits across two layers: the library owns lock state, legal transitions (pre-lock → hard-delete → write lock), and token checks, while each service decides only which of its operations are writes or deletes. Because the library is linked into every service, the safety boundary is uniform.
4. Lock Lifecycle
A migration progresses through four lock phases with escalating severity [5]. All phases use the same lock token issued during pre-lock.
Table 1.
Lock phases, configured expiry, and enforcement. Expiry is a liveness timeout; actual hold times are shorter. Reads are never blocked.
Table 1.
Lock phases, configured expiry, and enforcement. Expiry is a liveness timeout; actual hold times are shorter. Reads are never blocked.
| Lock Phase | Configured Expiry | User Impact | Enforcement Behavior |
|---|---|---|---|
| Pre-lock | 48h + convergence delay | None | Prevents concurrent migration; marks routing dirty |
| Hard-delete lock | 6h (renewable) | None visible | Service converts physical deletes to tombstones |
| Write lock | 5 min | Writes return retriable errors | Service blocks write operations |
| Swap + unlock | Instantaneous | None | Single atomic database write |
4.1. Phase 1: Pre-Lock with Passive Cache Convergence
The orchestrator requests a pre-lock (typically 48 hours). The service generates a lock token, marks the document migration-in-progress, and computes a convergence delay = maximum cache TTL across its tiers + database replication bound, returning the token, that delay (as the lock’s start time), and an expiry. The orchestrator then waits that delay plus a ∼10-minute safety buffer on a durable timer, and never less than the replication bound. The total varies by service (about 1–4 hours) and ensures that before stronger locks begin, every cache tier has expired its stale entry and re-fetched the now-dirty document.
Figure 2.
Passive cache convergence. After pre-lock, the orchestrator waits out the maximum cache TTL, the bounded replication lag (∼5 min), and a ∼10-minute safety buffer; all caches then re-fetch the dirty-flagged document with no active invalidation.
Figure 2.
Passive cache convergence. After pre-lock, the orchestrator waits out the maximum cache TTL, the bounded replication lag (∼5 min), and a ∼10-minute safety buffer; all caches then re-fetch the dirty-flagged document with no active invalidation.

Why passive convergence, not active invalidation? At Internet scale no one can enumerate every cache instance across clusters and edge and CDN tiers, and an invalidation that misses one could admit a write to the source during bulk copy. Passive expiry depends only on configuration: after the maximum TTL plus a buffer, every cache has converged, regardless of topology. We accept 1–4 hours of latency in this zero-user-impact phase for that guarantee, a bounded-staleness tradeoff in Vogels’s sense [11].
How the timing values were chosen. The replication bound is a per-service setting that trades freshness against cost: a tighter bound raises the store’s cost. Our chat location database uses 5 minutes, chosen through extensive production experimentation to keep cost relatively low; it is an example, not a design constant. The ∼10-minute buffer is engineering judgment, not an optimum. At about twice the replication bound, it absorbs requests in flight when the document was marked dirty, seconds of clock skew between the service that computes the start time and the orchestrator that waits on it, and durable-timer jitter. It costs at most about 15% of a 1–4-hour phase with no user impact, and monitoring (Section 5.4) has not shown it to be insufficient.
4.2. Phases 2–3: Hard-Delete and Write Locks
The orchestrator escalates to a hard-delete lock for the historical copy, then a brief write lock for the delta catch-up, both under the same token; the service-side behavior—tombstoned deletes, and retriable HTTP 423 (Locked) errors that client SDKs retry transparently—is detailed in companion work. [5] No lock can outlive the pre-lock, the parent lease bounding the whole migration. The orchestrator renews the hard-delete lock periodically, and the service caps the pre-lock duration, so an overrunning migration fails for investigation rather than renewing indefinitely.
4.3. Phase 4: Atomic Swap + Unlock
The orchestrator calls the swap API, and the service performs a single ETag-conditional point-write (Listing 1) that atomically: (a) updates locationInfo to the target region; (b) clears lockInfo, releasing all locks at once; and (c) locks the secondary location to prevent premature writes to the target before caches converge on it.
Because location and lock state share one document, partial cutover is impossible by construction: if the write succeeds, both changed; if it fails, neither did. This was a day-one design decision, not a post-incident fix, and it eliminates a failure class inherent to storing lock and routing state separately.
5. Evaluation
5.1. Methodology and Deployment Status
We evaluate the production deployment using 30 days of telemetry aggregated across regions, combining reliability data (Section 5.2), a latency analysis (Section 5.3), and a failure-mode analysis under the environment’s continuous, uncontrolled fault load (Section 5.4).
5.2. Reliability and Scale
The system processes 6–10 million migrations per day. The write-lock phase—the only user-impacting phase—holds for a median of 13 seconds (p75: 15s, p99: 19s), well under its 5-minute expiry. First-attempt success is 99.9%, and the remainder succeed on automatic retry. Over the 30-day window we observed no partial cutovers and no orphaned locks, and fewer than 0.1% of migrations outlived their lock and were retried with a fresh one. Migration-level results are reported in companion work [5].
All participating resource-owning services (the number has nearly doubled since launch) use the shared routing library, and none has implemented custom lock-enforcement logic.
5.3. Latency: Measured Cost, Inferred Comparison
What we measure. Our only lock-specific read is the read-through in Listing 1: a cached record carrying an active, started lock triggers one point read of the location document from the write region (2 request units), which service-side telemetry shows at ∼50 ms p99—the in-region Cosmos DB read itself is under 10 ms p99 per its SLA, [3] the rest network latency to the write region and service-side processing. Without such a lock, no lock-specific read occurs.
What we infer. Routing-embedded locking was this platform’s design from launch, so no earlier lock service exists to measure against, and deploying one on the live request path solely as a benchmark was not feasible. The read-through is, however, an in-situ proxy for the closest realistic baseline—a lock record in a dedicated table of the same store, as in the DynamoDB Lock Client: [12] the same point read against the same store, write region, and client stack. That baseline pays this read on every operation that might touch a migrating resource, since the service cannot know in advance which do; ours pays it only for resources already holding a lock. If a fraction f of operations target locked resources, the baseline performs about times as many lock reads; the two converge as f approaches 1. As an illustration only, if 1% of operations targeted locked resources, the baseline would perform about a hundred times as many lock reads as our design. We do not estimate the cost for consensus-based services, which add a hop to a separate cluster.
We therefore claim a structural reduction in critical-path lock reads whose per-read cost is measured. We do not claim a measured end-to-end latency or resource-usage improvement over a deployed alternative, which this evaluation cannot provide (Section 5.6).
5.4. Correctness Under Failure
Correctness rests on one invariant enforced inside the routing library: every mutation of the lock state is a single conditional write presenting both the current lock token and the document’s ETag. Only the token holder can advance or release the lock, and a new token is issued only after the previous lease expires or is released. Table 2 traces how this invariant, with lease expiry and the durable-workflow model, handles the failure classes a distributed lock must survive.
What five minutes of staleness does—and does not—allow. A replica may serve lock state that is stale by up to the configured bound—5 minutes here. That affects visibility—when a reader learns of a lock—not exclusion. Every acquisition is a write evaluated in the single write region against the current ETag; a writer acting on a stale read presents an outdated ETag, its write fails (412), and once its re-read reflects the live lock it is refused (423). Staleness can thus cause an orchestrator to attempt a conflicting acquisition, never to complete one. The visibility gap is closed in two ways. Pre-lock restricts no user operation until the convergence wait, which covers the replication bound, has elapsed. And once a document carries a lock, the routing library reads it only from the write region, never a replica, so enforcement never acts on lagging lock state. The orchestrator’s confirmation of each new lock is only an extra safeguard. One benign race remains: an orchestrator whose location read was stale may pre-lock a resource another instance has just swapped and released, so it re-reads the location after pre-lock and drops resources already at the target.
Is Table 2 exhaustive? Not formally. Its rows cover the faults admitted by the model in Section 2.5—crash, failover, message loss and partition, concurrency, staleness, lease timing, and version skew. Faults outside that model are not covered: Byzantine behavior, a bug in the routing library itself, the store violating its ETag or staleness guarantees, and operator force-release (Section 8), which is guarded procedurally by multi-level approval.
How correctness was checked. We have not applied formal methods such as TLA+; a machine-checked model of the token+ETag protocol is future work. Validation is empirical. Before release, each lock type was placed on test resources under generated production-style read, write, and delete traffic, checking that every operation was allowed, blocked, or tombstoned as the phase prescribes. In production, continuous synthetic probes repeat the cycle—acquire a lock, wait out convergence, confirm the lock on the read path, drive simulated reads and writes, and assert the outcomes—so an enforcement regression surfaces as a probe failure rather than a data anomaly. Testing shows correct behavior on the cases exercised, not the absence of bugs; confining safety to one invariant keeps that case space small.
Validating the convergence bound. Passive convergence is the one property that depends on external bounds rather than the token/ETag invariant. Each service declares its cache expiry plus replication bound (Section 4.1), the store enforces the replication ceiling, and the orchestrator monitors the declared delay against the actual pre-lock-to-hard-delete interval on every migration; across the window it stayed within bound, with no stale-cache write to a source region detected.
5.5. Architectural Comparison
Table 3 compares routing-embedded locking with established approaches on the properties that motivated the design. The comparison is structural, not a benchmark.
5.6. Threats to Validity
All metrics come from one platform. The comparisons in Section 5.3 and Section 5.5 are inferred and architectural, not head-to-head benchmarks; no before/after comparison exists because no earlier lock service ran here, and operational benefits are structural, not quantified against a counterfactual. The failure analysis rests on telemetry and testing, not controlled fault injection or formal verification. Finally, the shared library is a single point of correctness, mitigated by its narrow API, compatibility testing, and staged rollout.
6. Discussion
6.1. Applicability Beyond Messaging
The pattern applies when three conditions hold: a per-entity routing record is already resolved on every request; the entity to be locked is the entity that record routes, or a separate mechanism is needed; and cutover must change location and release locks together. Content delivery (per-asset routing), multi-tenant SaaS (per-tenant shard routing), and e-commerce (per-merchant regional routing) meet all three.
6.2. Infrastructure-Embedded Coordination as a Design Principle
Before building new coordination infrastructure, examine what already sits on the critical path—DNS, service-mesh routing, tenant resolution, shard mapping. This follows Helland’s case for entity-level rather than global coordination. [16]
6.3. Where the Complexity Goes
Removing a separate lock service relocates, rather than removes, the complexity of locking. The routing library now owns lock transitions, token validation, and the atomic swap for every service that links it; routing and locking anomalies can share a root cause; and the routing store inherits lock traffic (Section 8).
We accept this for three reasons. Routing and locking already share an availability requirement, consistency model, and operational team. Concentrating enforcement turns “every service must implement locking correctly” into “one audited component does.” And the library’s API is narrow, its token/ETag invariant unchanged since launch, and its lock API testable on its own, as the synthetic probes show (Section 5.4). Where routing and locking differ in consistency needs, owners, or lifecycles, a separate lock service remains the better choice.
6.4. 6.4 Alternatives Considered
Two alternatives deserve mention. The first is piggybacking coordination on an explicit lock acquisition, a standard technique for eliminating round-trips. Our design applies the same idea one step earlier—to the routing read that every request already performs—so even operations that never acquire a lock observe lock state without a separate lock lookup. The second, delegating migration to the underlying store, is set aside in Section 2.6.
7. Related Work
Consensus-based lock services. Chubby [1] and ZooKeeper [2] are standalone coordination services; we instead embed coordination state in infrastructure already on the request path.
Redis-based distributed locks. Redlock [13] depends on timing assumptions that clock skew and pauses violate; [15] our tokens are validated against stored state on every mutation.
Fencing tokens. Kleppmann [10] recommends fencing tokens that downstream services must validate. Our lock token realizes this pattern, with the check embedded in the shared library rather than left to each caller (§3.3).
Leases and database-backed locks. Gray and Cheriton [14] introduced the lease, a time-bounded grant whose lapse costs performance, not correctness. Our lock expiry is a lease in this sense, and passive convergence is a lease-style argument applied to caches. The DynamoDB Lock Client [12] is the closest analog—database-backed, lease-based, with heartbeat renewal. What differs is where the lease lives. A conventional database lease or conditional-write lock keeps lock state in a dedicated record, so verifying it costs a separate read. Ours lives in the routing document every request already reads, so verification needs no extra read unless a lock is active, and cutover and release are one atomic write.
Live migration. Pre-copy VM migration, [17] Albatross, [8] and Nomad [7] each operate within one storage system’s authority; our problem is coordinating locks and cutover across independently owned services (Section 2.6). Companion work surveys locks, leases, migration, and saga orchestration more broadly [5].
Companion work. Two companion articles, available as preprints, cover the surrounding system: the four-phase migration lifecycle [5] and the contract-based orchestration that invokes the lock and swap operations. [4] The lock mechanism, its correctness argument, and its evaluation are self-contained here.
8. Lessons Learned
Design lock durations for the failure case. Our initial fixed 7–10 day pre-lock, with no renewal, left resources stuck for days when a migration bug caused failures—the safety mechanism became the problem. Shorter 48-hour leases with bounded renewal (Section 4), plus admin tooling with multi-level approval to force-release stuck locks, were the response. The duration that matters is how long a failed migration blocks retries.
Extend existing infrastructure before building new infrastructure. Adding lock operations to the routing schema and library, which every service already used, gave us enforcement, atomic cutover, and uniform safety with no new infrastructure or on-call burden—in our experience far cheaper than operating a lock service, though we have not measured the difference (Section 5.6).
Anticipate read amplification in shared databases. Large migrations make millions of orchestrator calls, each reading a location document, which created partition hotspots that slowed unrelated resources. Because the lock makes the orchestrator the sole writer, the service can safely reuse a cached document whose ETag matches, cutting routing-database reads during migration by about 90%.
9. Conclusions
Routing infrastructure already on every request path can carry lock metadata, so enforcement needs no separate service and, for resources without an active lock, no additional lookup. Co-locating lock and location state yields routing-path enforcement, atomic cutover, and uniform safety; in production the write lock holds under 20 seconds at p99 across 6–10 million daily migrations, with no partial cutovers. The reduction in lock reads relative to a lock-service baseline is structural and inferred from a measured proxy, not benchmarked head-to-head. The pattern applies wherever per-entity routing records are resolved on every request and locking is needed at the same granularity.
Acknowledgments
The authors used GitHub Copilot to assist with drafting and editing; all technical content, design, and production data are the authors’ own, and the authors take full responsibility for the content. Aggregated, non-identifiable production metrics are shared with organizational approval.
Conflicts of Interest
All authors are employees of Microsoft Corporation, which operates the production system described in this paper. The authors declare no other conflicts of interest.
References
- M. Burrows, “The Chubby lock service for loosely-coupled distributed systems,” in Proc. 7th USENIX Symp. Operating Syst. Des. Implement. (OSDI), Seattle, WA, USA, Nov. 2006, pp. 335–350.
- P. Hunt, M. Konar, F. P. Junqueira, and B. Reed, “ZooKeeper: Wait-free coordination for Internet-scale systems,” in Proc. USENIX Annu. Tech. Conf. (ATC), Boston, MA, USA, Jun. 2010, pp. 145–158.
- Microsoft, “Consistency levels in Azure Cosmos DB,” Microsoft Learn. Accessed: Oct. 8, 2026. [Online]. Available: https://learn.microsoft.com/azure/cosmos-db/consistency-levels.
- D. Roy, S. Mulye, N. Govindarajan, and J. Rajeevalochanam, “Moving live data across the continuum: Contract-based migration when no service knows the full graph,” submitted to IEEE Softw., 2026. Preprint, doi: 10.20944/preprints202610.0226.v1. [Online]. Available: https://www.preprints.org/manuscript/202610.0226.
- D. Roy, S. Mulye, N. Govindarajan, and J. Rajeevalochanam, “Live cross-region data migration without perceived downtime: A four-phase protocol for multi-service cloud systems,” submitted to IEEE Access, 2026. Preprint, doi: 10.20944/preprints202610.0828.v1. [Online]. Available: https://www.preprints.org/manuscript/202610.0828.
- S. Singh, “Powering real-time messaging at scale with Azure Cosmos DB,” Azure Cosmos DB Blog, 2025. Accessed: Oct. 8, 2026. [Online]. Available: https://devblogs.microsoft.com/cosmosdb/powering-real-time-messaging-at-scale-with-azure-cosmos-db/.
- N. Tran, M. K. Aguilera, and M. Balakrishnan, “Online migration for geo-distributed storage systems,” in Proc. USENIX Annu. Tech. Conf. (ATC), Portland, OR, USA, Jun. 2011.
- S. Das, S. Nishimura, D. Agrawal, and A. El Abbadi, “Albatross: Lightweight elasticity in shared storage databases for the cloud using live data migration,” Proc. VLDB Endow., vol. 4, no. 8, pp. 494–505, May 2011.
- Microsoft, “Database transactions and optimistic concurrency control in Azure Cosmos DB,” Microsoft Learn. Accessed: Oct. 8, 2026. [Online]. Available: https://learn.microsoft.com/azure/cosmos-db/database-transactions-optimistic-concurrency.
- M. Kleppmann, Designing Data-Intensive Applications. Sebastopol, CA, USA: O’Reilly, 2017, ch. 8.
- W. Vogels, “Eventually consistent,” Commun. ACM, vol. 52, no. 1, pp. 40–44, Jan. 2009.
- Available online: https://github.com/awslabs/amazon-dynamodb-lock-client (accessed on Oct. 8 2026).
- S. Sanfilippo, “Distributed locks with Redis,” Redis Documentation. Accessed: Oct. 8, 2026. [Online]. Available: https://redis.io/docs/latest/manual/patterns/distributed-locks/.
- C. G. Gray and D. R. Cheriton, “Leases: An efficient fault-tolerant mechanism for distributed file cache consistency,” in Proc. 12th ACM Symp. Operating Syst. Princ. (SOSP), Litchfield Park, AZ, USA, Dec. 1989, pp. 202–210.
- M. Kleppmann, “How to do distributed locking,” Feb. 8, 2016. Accessed: Oct. 8, 2026. [Online]. Available: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html.
- P. Helland, “Life beyond distributed transactions: An apostate’s opinion,” in Proc. 3rd Biennial Conf. Innov. Data Syst. Res. (CIDR), Asilomar, CA, USA, Jan. 2007, pp. 132–141.
- C. Clark et al., “Live migration of virtual machines,” in Proc. 2nd USENIX Symp. Netw. Syst. Des. Implement. (NSDI), Boston, MA, USA, May 2005, pp. 273–286.
Figure 1.
The location document stores routing and lock state in the same database document. The primary section contains the active location and any migration locks. The secondary section is created during migration to hold the target location and prevent premature writes to the target.
Figure 1.
The location document stores routing and lock state in the same database document. The primary section contains the active location and any migration locks. The secondary section is created during migration to hold the target location and prevent premature writes to the target.

Table 2.
Failure classes and the properties that handle them, corroborated by production telemetry and testing rather than controlled fault injection. Host loss, deployments, and failovers occur continuously in our environment, so 30 days without a partial cutover is sustained evidence.
Table 2.
Failure classes and the properties that handle them, corroborated by production telemetry and testing rather than controlled fault injection. Host loss, deployments, and failovers occur continuously in our environment, so 30 days without a partial cutover is sustained evidence.
| Failure | Guaranteed behavior | Why |
|---|---|---|
| Orchestrator crash mid-migration | Migration resumes with no manual action; impact bounded by lease | The orchestrator is a durable workflow that resumes on any of 300+ machines from its last checkpoint, token persisted. A crash during the write lock lets the short lease auto-release, so writes resume. |
| Database failure / regional failover | No partial cutover; migration retries | Cutover is one atomic conditional write. The durable workflow retries transient failures against a healthy write region. Lock durability equals routing durability; no separate store can desynchronize. |
| Network partition / split-brain | No two writers can both mutate a lock | The replicated store, not an application node, holds the record. A mutation needs the matching token and ETag in one compare-and-swap, so a second writer loses and retries. |
| Concurrent migration attempts | Exactly one proceeds; others abort cleanly | Pre-lock is itself exclusive: a request finding an unexpired lock under another token gets HTTP 423; of two simultaneous requests, the loser’s ETag-conditional write fails (412) and its re-read finds the winner’s lock. The loser aborts before any destructive action. Observed at <0.1% of migrations, failing as designed. |
| Stale cache during bulk copy | No write reaches the source after its lock is in force | The convergence wait (§4.1) exceeds the declared maximum cache TTL plus the bounded replication lag, so every cache tier has re-read the locked document before stronger locks begin. |
| Lease expiry before completion | No stale holder can act; resource becomes migratable again | An expired lease invalidates its token; a slow orchestrator that resumes finds its token rejected. [10] A fresh migration re-drives from a consistent source. |
| Mixed service versions during rollout | No version can bypass enforcement | The token+ETag invariant lives entirely in the shared library and has been stable since launch; compatibility tests and a schemaVersion field make rollout safe. |
Table 3.
Comparison with distributed lock approaches. A dedicated lock record can also be updated atomically; what differs is that our swap-and-release acts on the routing document already read on every request.
Table 3.
Comparison with distributed lock approaches. A dedicated lock record can also be updated atomically; what differs is that our swap-and-release acts on the routing document already read on every request.
| Property | Routing-Embedded (Ours) | Chubby [1] | ZooKeeper [2] | Redlock [13] | DynamoDB Lock [12] | Generic DB lease [14] |
|---|---|---|---|---|---|---|
| Additional infrastructure | None | Paxos cells | ZK ensemble | Redis instances | Dedicated table | Dedicated table |
| Lock-check latency | No extra read unless a lock is active; then one point read | Separate RPC | Separate RPC | Separate call | Separate read | Separate read |
| Atomic swap + unlock | Yes—in the routing document already on the read path | No | No | No | Only via a separate lock record | Only via a separate lock record |
| Fencing token | Yes (library-enforced) | Yes (sequencer) | Must build | Controversial [15] | Yes (record version) | Via version |
| Clock assumption | None (token/ETag) | None | None | Bounded skew [15] | None | Bounded (lease) |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.