From Monolith to Distributed: How a Demo Project Tackles Overselling, Duplicate Orders, and Data Consistency Stage by Stage
As a developer who works mainly with React / Node.js, I wanted to validate hands-on how the Spring ecosystem achieves distributed transactions and eventual consistency. spring-order is the demo project I built for that purpose (spring-order, see erishen/spring-order): the initial architecture is a standard monolith — MySQL stores orders and inventory, Spring transactions guarantee consistency, and the React frontend calls the REST APIs. Overselling, duplicate orders, and message loss — these classic challenges of distributed order systems are exactly what the project tackles, stage by stage.
Below is the step-by-step process of how I evolved and hardened the system. Each phase corresponds to a class of representative problems.
TL;DR
spring-order is a Spring Boot + React order-system demo that evolves stage by stage, P0→P4: P0 uses @Version optimistic locking to prevent overselling; P1 introduces a Redis distributed lock (SET NX + Lua atomic unlock) and cache to serialize inventory deductions across instances; P2 uses Kafka transactional Outbox to put "create order" and "emit event" in the same local transaction, so messages are never lost; P3 adds Resilience4j rate limiting, circuit breaking, and retry to absorb downstream jitter and traffic spikes; P4 adds nginx multi-instance + Prometheus/Grafana observability. Every stage targets the same goal — no overselling, no duplicate orders, and eventual consistency under high concurrency.
P0: Monolith + Optimistic Locking — The First Pitfall Was Overselling
In the earliest version, inventory deduction relied on @Version optimistic locking. The logic was straightforward: query the inventory, check the quantity, include the version number on update, and retry on CAS failure.
inventoryMapper.updateById(inventory)
This logic was more than sufficient at tens of QPS. But under concurrent load, dozens of requests read the same product's "sufficient inventory" simultaneously, then each submitted their own update — @Version only guarantees concurrency safety for a single record, not a global inventory cap across requests. At the database level, multiple records were updated in parallel; version conflicts eliminated only some of them, while the rest went through, and inventory was deducted straight into negative numbers.
I realized that optimistic locking in this scenario was essentially "last writer wins," not "inventory cannot exceed the upper limit."
The fix was to serialize inventory deductions. But serialization can't rely on database row locks — the throughput couldn't handle it. I needed a mutual exclusion mechanism that multiple service instances could share.
P1: Redis Distributed Lock + Cache — Cross-Instance Serialization
I implemented a distributed lock using Redis SET NX. The core logic: before deducting inventory, acquire a lock on the product ID first, and execute the deduction operations serially while holding the lock.
At the same time, I migrated the frequently read inventory data from MySQL to a Redis cache layer, avoiding every deduction penetrating all the way to the database. Reads went through the cache; deductions were backed by the dual guarantee of lock + DB.
This approach solved the overselling problem, but it also brought new challenges: what if the lock times out? If the instance holding the lock crashed, the lock would never be released, and all other requests would block. For unlock I use a Lua script — it only runs del when the currently held random token matches, so an already-expired lock is never mistakenly released by another instance. The project does not depend on Redisson; it is a hand-written SET NX + Lua unlock. When the lock cannot be acquired in time, it degrades to direct execution backed by the @Version optimistic lock.
But distributed locks themselves introduced a new problem — idempotency.
P2: Idempotency Service + Resilience4j — Duplicate Requests and Cascade Failure
After overselling was fixed, the next typical problem to confront is duplicate orders. The usual triggers are clients retrying on network lag, or the frontend component not disabling its button immediately after submission — requests hit the backend multiple times, each passing the lock check, so two orders get created.
I had clients send an Idempotency-Key request header on each order call; the backend stores it in a database table (idempotency record) and applies three-way handling: already completed → replay the previous response, in progress → return a 409 conflict, this request owns it → proceed. That way a duplicate submission with the same key only ever creates one real order.
Meanwhile, I noticed the system's fragility — when a downstream service (such as the inventory service) responded slowly, the caller's threads would pile up, eventually bringing down the entire service. I added Resilience4j's three-layer protection:
@RateLimiter: Limits the number of calls per unit time, preventing sudden traffic spikes from overwhelming downstream services@CircuitBreaker: Opens the circuit when the failure rate exceeds a threshold, failing fast instead of retrying indefinitely@Retry: Performs a limited number of automatic retries for transient jitter
These three layers of protection let me see "the circuit breaker tripped" on the dashboard for the first time, instead of "the thread pool is full." System observability shifted from "hindsight" to "early warning."
But a new architectural problem emerged: data consistency between order creation and inventory deduction.
P3: Kafka Outbox — Cross-Service Data Consistency
In the P2 phase, order creation and inventory deduction were completed within the same transaction — no problem in a monolith. But when I split the order service and inventory service apart, a local transaction could no longer span both services. Using two-phase commit (2PC) would drastically increase system complexity, and the introduction of Kafka made things even more subtle.
My solution was the Outbox pattern. Within the order creation transaction, I wrote to both the order table and the outbox table simultaneously, then a service was responsible for publishing outbox records to Kafka — polling the outbox table and batching un-sent events.
After subscribing to Kafka events, the inventory service executed deductions, guaranteeing exactly-once processing through idempotency keys (message IDs). Even if the service restarted, no events would be lost — because the outbox table itself is a persistent "pending send queue."
The core benefit of this approach: order creation and outbox writes happen within the same local transaction, ensuring "either both are written, or neither is." Inventory deduction, though in another service, achieves eventual consistency through reliable Kafka event delivery and idempotent processing.
The cost was introducing a Kafka dependency and polling logic. I chose this cost because data consistency is a non-negotiable baseline.
P4: Multi-Instance Deployment + Gateway + Observability — The Final Puzzle
P3 solved the data consistency problem in a distributed environment, but the system still needed to support horizontal scaling. I deployed multiple order service instances with nginx in front for load balancing. I also enabled Actuator metrics exposure on each instance, integrating Prometheus and Grafana.
Now I could see not just "is the system up or down," but:
- QPS and P99 latency for each endpoint
- Circuit breaker status and trip counts
- Kafka consumer lag
- Redis cache hit rate
These metrics allowed me to sense trending changes before problems occurred, rather than waiting for failures to surface before troubleshooting.
Evolution Roadmap Summary
Looking back at the entire evolution, my core approach never changed: solve representative problems first, then consider architectural elegance.
| Phase | Problem | Solution |
|---|---|---|
| P0 | Overselling | @Version optimistic locking (insufficient) |
| P1 | Cross-instance concurrency | Redis distributed lock + cache |
| P2 | Duplicate orders, cascade failure | Idempotency service + Resilience4j |
| P3 | Cross-service data consistency | Kafka Outbox pattern |
| P4 | Insufficient observability | Multi-instance + nginx + Prometheus |
Each phase was a closed loop of "problem occurs → root cause identified → solution introduced → effect verified." There is no one-step-perfect architecture — only continuous iteration toward correctness.
Source Code Navigation
- Order endpoint & optimistic lock:
OrderService.java(@Version) - Redis distributed lock (SET NX + Lua atomic unlock):
RedisLockService.java - Idempotency key (
Idempotency-Keyheader + DB store):IdempotencyService.java/OrderController.java - Kafka transactional Outbox (event & business in same transaction):
OrderService.java(saveEvent) - Rate limit / circuit break / retry:
application.yml+GlobalExceptionHandler.java - Multi-instance gateway & observability:
docker-compose.yml/prometheus.yml/grafana/
Why can't optimistic locking solve the overselling problem?
Optimistic locking only guarantees version number conflict detection for a single record. Multiple concurrent requests can read the same inventory value simultaneously and each pass the validation, resulting in over-submission.
How does the Redis distributed lock prevent lock loss?
It acquires the lock with Redis SET key token NX PX (default 3-second TTL), where token is a fresh random UUID each time. Unlock runs a Lua script that only executes del when the held token still matches — so even an expired lock is never released by the wrong instance. If the lock can’t be acquired, it degrades to direct execution guarded by the DB @Version optimistic lock (more retries, but no overselling). The project does not use Redisson; it is a hand-written SET NX + Lua unlock.
What advantages does the Outbox pattern have over sending Kafka events directly?
Outbox places event writing and business operations within the same local transaction, ensuring “order creation succeeds if and only if the outbox record is written successfully,” avoiding the inconsistent state where the business succeeds but the event is lost.
How is the idempotency key designed?
The key is sent by the client via the Idempotency-Key request header (not generated server-side), and the backend stores it in a database table via checkAndReserve(key). Three outcomes: HIT means a prior attempt already completed → replay the cached response (200); IN_PROGRESS means another request with the same key is in flight → return 409 conflict; PROCEED means this request owns the key. On success complete(key) persists the response; on failure fail(key) releases it so the client can safely retry with the same key. Concurrent inserts are guarded by a unique index.
What problems does each of Resilience4j's three layers of protection solve?
RateLimiter limits sudden traffic spikes, CircuitBreaker fails fast when downstream services are faulty to prevent cascade failure, and Retry performs a limited number of automatic retries to recover from transient jitter.
What metrics does the observability introduced in P4 primarily focus on?
Endpoint QPS and P99 latency, circuit breaker status and trip counts, Kafka consumer lag, and Redis cache hit rate — these metrics support proactive trend awareness rather than reactive troubleshooting.
Project
- GitHub repository: https://github.com/erishen/spring-order
- Demonstrates the P0–P4 architecture evolution, transactional Outbox, idempotency, Resilience4j rate limiting / circuit breaking / retry, and the nginx multi-instance gateway + Prometheus/Grafana observability.