A Lesson From Nine Global Business Units
This is Part 2 of "The Developer Advocate's Journey" series. Before I ever carried a DevRel title, I led an engineering team through a mistake that taught me more about trust than any deprecation notice or launch announcement ever could.
What you'll learn:
- 🚨 Why "regional autonomy" can quietly become "regional chaos" at scale
- 🛠️ The architecture decision that caused it, and the one that fixed it
- 📋 The principles I now design distributed systems by
- 📈 What actually happened when we rebuilt it properly
The Setup: Nine Business Units, One Assumption#
In September 2018, I joined Nissan Digital India as a Lead Software Engineer, inheriting a team of 12 engineers responsible for systems supporting 9 global Sales Finance Business Units (SFBUs) spread across multiple regions.
Each SFBU ran its own local copy of customer records, synced to the others on a nightly batch job. It wasn't a careless decision - it was a reasonable one, made under real constraints:
- Regional teams wanted autonomy over their own data and workflows
- A real-time, globally consistent sync layer sounded like months of extra engineering before we could ship anything
- Nightly batch sync had "worked fine" in the early pilot regions
That assumption is the one that came back to haunt us.
When Reality Caught Up#
As more SFBUs came online, small differences between regions - phone number formats, address fields, naming conventions, which system was allowed to be the "source of truth" for a given customer - started to compound instead of average out.
The nightly batch job, which was supposed to reconcile records, often did the opposite: a simple "last write wins" rule would silently overwrite a correct, recently-updated record in one region with a stale one from another. The same customer could look different depending on which business unit had touched their record most recently.
Regional finance and customer service teams started noticing during month-end reconciliation and everyday customer interactions. Within a couple of quarters, more than one of the nine SFBUs had raised the same concern independently: which record was actually correct?
How a 'Good Enough' Assumption Became a Real Problem
Formatting drift, dismissed as noise
Minor mismatches between regional records surfaced first - easy to explain away as expected regional variance.
Key Achievements:
- ●Different address and phone formats across regions
- ●Occasional duplicate customer entries after batch sync
- ●Treated as data hygiene, not architecture
'Last write wins' started erasing correct data
As more SFBUs onboarded, the nightly batch sync began overwriting good records with stale ones from other regions.
Key Achievements:
- ●No single business unit was reliably the 'source of truth'
- ●Conflicts increased faster than the number of regions
- ●Manual fixes became a recurring, unplanned workload
Multiple business units raise the same red flag
Finance and customer service leads across more than one SFBU independently questioned which customer record was authoritative.
Key Achievements:
- ●Month-end reconciliation surfaced conflicting records
- ●Customer service reps saw different details depending on region
- ●Leadership acknowledged the federated model couldn't hold at this scale
We committed to rebuilding the foundation
Instead of patching the batch job again, we scoped a real Customer Master Data Management platform.
Key Achievements:
- ●Agreed a golden-record strategy was non-negotiable
- ●Prioritized event-driven sync over nightly batch
- ●Set data governance as a first-class requirement, not an afterthought
The Hard Lessons: What I Learned#
1. Regional Autonomy Without a Golden Record Doesn't Scale#
What I thought: Letting each business unit own its local data would keep things simple and fast to ship. Reality: Past a handful of regions, "who owns the truth" becomes the hardest question in the system - and nightly batch sync has no good answer for it.
The fix: A single master data repository with a clear golden-record strategy, so every region reads from and writes to the same source of truth.
2. Conflict Resolution Has to Be a Business Decision, Not a Technical Default#
"Last write wins" is a technical shortcut, not a business rule. It quietly encodes "whoever synced most recently is right," which is almost never true.
The fix: Conflict resolution logic that reflects actual business priority - which system, which field, which region takes precedence, and why - built and reviewed with the business stakeholders who understand the data.
3. Compliance and Governance Belong in the Architecture, Not Bolted on Later#
Operating across multiple regions meant GDPR and other regional data protection rules were always in scope, even when the original design treated them as a later concern.
The fix: Build data governance, audit trails, and regional compliance rules into the platform from day one.
4. Distributed Systems Need Real Monitoring, Not Just Batch Logs#
A nightly job that "completes successfully" can still be actively corrupting data. Without real-time visibility into sync health, we found out about problems from finance teams instead of dashboards.
The fix: Continuous monitoring and observability for every synchronization path, so anomalies surface before a business unit has to report them.
The Recovery: Building CMDM#
The response wasn't a quick patch - it was a proper platform: Customer Master Data Management (CMDM), built over roughly 18 months using an event-driven microservices architecture on Apache Kafka and Spring Boot, with a golden-record data model and a CQRS pattern separating read and write paths.
What the Rebuild Actually Delivered
Data Inconsistencies
Reduction in customer-data conflicts across regions
Regional Coverage
Regional operations unified on one master data platform
Service Resolution
Faster customer service response times
Annual Savings
Operational efficiency from eliminating duplicate/conflicting data work
Data Consistency
Consistency maintained across all regions post-launch
System Downtime
Reduction in downtime across the systems I was responsible for
The Framework: How I Design Distributed Data Systems Now#
The Golden Record Principles#
Federated Regional Model vs. Golden Record Model
Each region owns a local copy, reconciled nightly
✅ Pros:
- •Fast to ship initially
- •Regions feel ownership of their own data
- •No upfront investment in sync infrastructure
❌ Cons:
- •Conflicts compound as regions are added
- •No reliable source of truth
- •Batch sync hides problems until reconciliation
Event-driven sync into one master data repository
✅ Pros:
- •Single source of truth with intelligent conflict resolution
- •Sub-second propagation instead of overnight batches
- •Governance and compliance designed in from the start
❌ Cons:
- •Real upfront investment in event-driven architecture
- •Requires cross-regional alignment before launch
- •More moving parts to monitor and operate
We paired the golden-record model with an event-driven microservices architecture (loose coupling and independent scaling per region) and a CQRS pattern (so heavy analytics reads never competed with transactional writes). Neither choice was exotic - both were the direct answer to the specific failure mode we'd already lived through.
What This Taught Me About Trust#
Leading a distributed team isn't just about shipping features. It's about deciding, deliberately, who owns the truth when multiple teams touch the same data - and building the system so that decision is enforced automatically instead of discovered during a month-end close.
The trust we lost with those nine business units in the first year took real, sustained work to rebuild: honest acknowledgment of what had gone wrong, a credible rebuild plan, and results that regional teams could verify for themselves.
The Silver Lining#
This experience reshaped how I approach every system I've touched since:
- I question "regional autonomy" by default - it's often architecture debt wearing a friendly name
- I ask "who owns the truth" before writing a line of sync logic
- I treat compliance and monitoring as day-one requirements, not phase-two cleanup
- I bring business stakeholders into conflict-resolution design, not just engineering
Most importantly: admitting the original design was wrong, in front of the business units it affected, did more for trust than trying to quietly patch around it.
Your Turn: Avoiding the Same Mistake#
If you're designing a system that multiple teams or regions will touch:
- Decide who owns the truth before you decide how to sync it
- Never let "last write wins" be your actual conflict-resolution strategy
- Build compliance and monitoring in from day one, not after the first incident
- Loop in the business stakeholders who understand the data, not just the engineers moving it
- Have a plan for what happens when regions disagree - because they will
Lesson Learned
Thakur Ganeshsingh
Years later, this remains one of the most valuable lessons of my career: at scale, "who owns the truth" is an architecture decision, not a documentation footnote. Every system that lets multiple teams write to the same data either answers that question deliberately, or answers it by accident - usually at the worst possible time.
Next in the series: "0 to 10,000 Developers: The Community Building Playbook That Actually Works"
Have you lived through a similar rebuild? Share your story in the comments - the community learns from our collective mistakes.
Want more honest lessons from a decade across engineering and developer relations? Subscribe to get each new post in this series.
