Data Synchronization

Data synchronization is how the same business fact stays consistent across multiple systems. A customer order placed in the buyer’s EDI system has to become a sales order in AgriERP, a delivery commitment in the warehouse system, and (eventually) an invoice in the buyer’s accounts payable system. A grower payment captured in AgriERP has to become an instruction at the bank, a transaction in the bank’s ledger, a confirmation back in AgriERP, and a notification on the grower’s phone. Done well, all of this happens automatically, in real time, with no manual reconciliation. Done badly, businesses spend weeks each month trying to figure out why two systems disagree about the same fact.

This document describes how AgriERP handles data synchronization: real-time vs. batch patterns, bidirectional flows, change-detection mechanisms, conflict handling, and the operational practices that keep multiple systems aligned without constant manual intervention

1. Real-time synchronization

Where real-time matters

Capability	What it means in practice
Sensor and IoT data	Cold-store temperatures, equipment telemetry, weighbridge readings. A 30-minute delay in a cold-store alert is the difference between intervening and losing a load.
Sales orders from buyers	Orders arriving via EDI need to be visible in operations immediately, so dispatch planning can react. A 4-hour delay can mean a missed dispatch window.
Payment confirmations	Customer payments hitting the bank need to update AR and clear credit holds immediately, so the customer can place the next order without artificial blocks.
Grower settlements	Settlement payments to growers need confirmation flowing back in real time, so growers know they have been paid and the business knows the run completed.
Mobile field data	Work-order completions, harvest counts, and quality observations from the field need to flow into the office in real time so planning and dispatch reflect what is actually happening.
Inventory movements	Receipts, picks, putaways, and dispatches need to update inventory in real time so allocation decisions and shortage alerts use current data.

How real-time sync works

• Event-driven architecture: when something changes in one system, an event is published. Any system interested in that event consumes it and updates accordingly. No system has to ask, “has anything changed?” repeatedly.
• Message broker: events flow through a message broker that handles delivery, ordering, retry, and durability. Senders fire events; receivers process them; the broker handles the rest.
• Webhook pattern: for external systems that support webhooks (push notifications of changes), AgriERP exposes endpoints to receive them and reacts on arrival.
• Push from AgriERP: for outbound flows, AgriERP pushes changes to external systems as they happen, via REST API calls, message queues, or other channels supported by the partner.
• Streaming for high-frequency data: for sensor data and other high-volume continuous flows, streaming pipelines deliver data into AgriERP with sub-second latency where the use case requires it.

Real-time sync guarantees

• At-least-once delivery: every event is delivered, even if a single attempt fails. Combined with idempotency, this ensures no event is lost, even when network or system issues occur.
• Ordering where it matters: for flows where event order is important (sequence of inventory movements on the same lot), ordered delivery within the relevant partition is guaranteed.
• Latency targets: different flows have different latency targets. Cold-store alerts target sub-minute; sales orders target sub-five-minute; less critical flows have looser targets. Targets are monitored and reported.
• Replay on failure: where a downstream system is down for a period, events accumulate and are replayed when the system recovers. The history is not lost.

2. Batch and scheduled synchronization

Where batch makes sense

Capability	What it means in practice
Daily reference data	Currency exchange rates, market prices, weather summaries, daily fuel prices. Once a day is enough; real-time adds complexity without value.
Periodic statements	Bank statements, vendor statements, customer statements. Typically end-of-day or end-of-month exchanges.
Bulk loads	Initial data loads, year-end imports, large historical data transfers. Batch handles these better than streaming.
Tax filings and statutory submissions	Periodic returns and submissions, where the cadence is set by the regulator and there is no value in real-time.
Analytics warehouse loads	Daily or hourly refreshes of the analytics layer with operational data. Some real-time on top, but the bulk of the load is batch.
Master-data distribution	Pushing master data (item catalogs, customer lists) to downstream systems on a defined cadence.

Batch sync mechanics

• Scheduled jobs: batch jobs run on configured schedules, daily overnight, hourly, weekly. The schedule is part of the integration configuration.
• Change-data capture: rather than re-extracting the full dataset every time, batch jobs typically extract only what has changed since the last run. Last-modified timestamps and change-log tables drive incremental sync.
• Bulk operations: batch operations are designed for volume. Loading 50,000 records as a bulk operation is far more efficient than 50,000 individual API calls.
• Validation in bulk: batch ingestion can validate large datasets thoroughly: schema checks, business-rule checks, referential integrity, with detailed error reports for any rejected records.
• Checkpointing: long-running batch jobs checkpoint progress, so a failure mid-run can resume from the checkpoint rather than restarting from scratch.
• Reconciliation reports: every batch run produces a reconciliation report: records read, records written, records rejected, with reasons. Operations teams know whether the run was clean.

Mixing real-time and batch

Real-time and batch are not exclusive choices; most well-designed integrations use both. Sales orders may arrive in real time as buyers place them, but a nightly batch reconciliation compares all orders received with the buyer’s master record to catch anything that slipped through. Bank reconciliation matches transactions in real time as they arrive, with a daily catchup against the official end-of-day statement.

3. Bidirectional flow

Why bidirectional

A buyer order is the obvious case. The buyer’s PO flows in; AgriERP’s order confirmation flows back; an advance shipment notice flows out; the buyer’s receipt confirmation flows in; the invoice flows out; the payment flows in. The same business transaction (a single order fulfillment) involves multiple data movements in both directions across the integration. Bidirectional integration treats this as a coherent business flow, not as separate inbound and outbound integrations that happen to be related.

Standard bidirectional flows

Capability	What it means in practice
Order-to-cash with buyers	Inbound PO, outbound acknowledgement, outbound ASN, inbound receipt confirmation, outbound invoice, inbound payment. Each step has its own message; all are coordinated as a single order lifecycle.
Procure-to-pay with vendors	Outbound PO, inbound acknowledgement, inbound ASN, outbound receipt, inbound invoice, outbound payment. The mirror of order-to-cash.
Banking	Outbound payment files, inbound payment confirmations, inbound bank statements, outbound reconciliation queries. Continuous flow.
Carrier integration	Outbound shipment requests, inbound capacity acceptance, inbound tracking updates, inbound delivery confirmation, inbound freight invoice, outbound payment.
Master data synchronization	AgriERP item catalog flows out to a buyer EDI system; buyer’s identifier mapping flows back. Both systems carry the equivalent reference data with mappings between them.
Customs and tax authorities	Outbound declarations and filings, inbound assessments and confirmations, outbound payments, inbound clearance documents.

Avoiding ping-pong

• Source of truth: for any data field, one system is the source of truth. Other systems sync to it but do not write back changes that contradict it. Without this rule, sync becomes a war of who saved last.
• Field-level ownership: different fields on the same record can have different sources of truth. The buyer’s PO number is owned by the buyer; the internal sales order number is owned by AgriERP; the dispatch date is owned by AgriERP; the delivery date is owned by the carrier.
• Change-source tagging: every change carries a tag indicating its source. A change that flowed in from an integration is not echoed back to the source as if it were a local change.
• Update-detection: where a partner system pushes a full snapshot regularly, AgriERP compares against the last received snapshot to detect actual changes, rather than treating every push as new.

4. Conflict detection and resolution

Where conflicts come from

• Concurrent updates: two systems update the same record near-simultaneously. Even with clear ownership rules, in rare cases both sides change at once.
• Connectivity gaps: where one side has been offline (a field device, a remote office, a downed partner system), changes accumulate. When connectivity returns, multiple changes need to merge.
• Out-of-order delivery: in distributed systems, events can occasionally arrive out of order. The system has to detect when this happens and apply changes in their logical sequence, not their arrival sequence.
• Bad data on one side: a partner system sends a value that does not match what is known elsewhere (a customer’s address differs between AgriERP and the CRM). The conflict needs detection and resolution.
• Reference data drift: where reference data (item codes, location codes) gets out of sync between systems, downstream conflicts cascade. The integration needs to detect and flag the underlying drift.

Conflict resolution strategies

Capability	What it means in practice
Last-write-wins	Simplest strategy: the most recent change overwrites earlier changes. Works for data where the latest state is what matters and earlier states are transient (a current price, a current temperature).
Source-of-truth wins	The configured source of truth wins, regardless of timing. Useful where one system is authoritative (the bank is authoritative on the bank balance; AgriERP is authoritative on internal cost data).
Merge	Where fields can be merged (combining the additions to a list, applying both updates to different fields), the integration merges rather than choosing. Common for collaborative scenarios.
Human review	Where the conflict cannot be resolved automatically, it is surfaced to a human reviewer with the conflicting values and the context. The reviewer decides; the decision is captured and feeds future automation.
Block until resolved	For high-stakes conflicts (financial transactions, regulatory data), the integration blocks further changes until the conflict is resolved. Prevents the conflict from cascading downstream.

Operational practices for conflict-free sync

• Clear ownership: every data field has a clear source of truth, documented and enforced by the integration logic. Conflicts get rare.
• Validation at boundaries: inbound data is validated before it enters AgriERP. Outbound data is validated before it leaves. Bad data gets rejected at the boundary, not allowed to corrupt downstream systems.
• Idempotency everywhere: every integration operation can be safely retried without creating duplicates. The same message processed twice produces the same end state as processing it once.
• Reconciliation checks: periodic reconciliation jobs compare key data between systems and flag mismatches. The mismatches lead to root-cause investigation; the underlying causes get fixed.
• Audit trail: every sync action and every conflict decision is logged. Disputes about “who changed what when” have structured answers.

In summary

Data synchronization in AgriERP is the discipline that keeps multiple systems aligned about the same business facts. Real-time sync handles the flows where operational decisions depend on current data, using event-driven patterns through a message broker. Batch sync handles the flows where periodic alignment is enough, with change-data capture, bulk operations, and reconciliation reports. Bidirectional flow is the common case, with field-level ownership and source-of-truth rules avoiding ping-pong updates. Conflict detection and resolution handle the rare cases where systems disagree, with strategies ranging from last-write-wins through to human review.

Combined with the BURQ middleware that runs the integrations and the catalog of external systems they connect to, data synchronization is what makes the modern agribusiness operate as a coherent whole. Buyers, suppliers, banks, regulators, partners, and internal systems all stay aligned, automatically, in real time where it matters, with audit trail throughout.