From ad0e2773808dbb62439c1309ed96ef513d1b796d Mon Sep 17 00:00:00 2001 From: Gurveen Kaur Date: Tue, 24 Mar 2026 14:49:06 +0530 Subject: [PATCH 1/3] Add proposal for event-driven integration approach with D365 --- .../EVENT_DRIVEN_INTEGRATION_APPROACH.md | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md diff --git a/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md b/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md new file mode 100644 index 00000000..475de8bd --- /dev/null +++ b/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md @@ -0,0 +1,60 @@ +# Proposal: Event-Driven Integration (Outbox Pattern) for ERP Systems (D365/NetSuite) + +## Overview +This document proposes a standardized, decoupling-first approach to integrating Moqui's Order/Return/Shipment data with external ERP systems like Dynamics 365 and NetSuite. + +## The Problem +Many traditional integrations in Moqui either rely on manual "sweeps" (polling) of entity tables or complex `service-call` chains that happen synchronously during user actions. This leads to: +1. **Performance Bottlenecks**: High-latency API calls to external systems block UI threads or database transactions. +2. **Atomicity Issues**: If an integration service fails after a database change, the external system is out of sync. +3. **Difficult Monitoring**: No centralized "Integration Table" to track which events have been logged but not yet sent. + +## The Solution: Moqui "Outbox" Pattern +We propose leveraging Moqui's native `DataFeed`, `DataDocument`, and `SystemMessage` infrastructure to implement an asynchronous **Outbox Pattern**. + +### 1. Unified Event Detection (`DataFeed` & `DataDocument`) +Instead of polling `OrderHeader` for status changes, we define a set of `DataDocument`s that capture the core event data. + +- **DataDocument**: Lightweight structure containing the `orderId` and `statusId`. +- **DataFeed**: A real-time (`DtfdptRtPush`) monitor that triggers on entity changes. It ensures transactional integrity by only firing after a successful DB commit. + +### 2. Integration Event Log (`SystemMessage`) +When a `DataFeed` triggers, it calls a lightweight logging service that generates a `SystemMessage`. This record acts as our persistent "Integration Table". + +- **SystemMessageType**: Represents the event (e.g., `ORDER_CREATED`, `ORDER_CANCELLED`, `SHIPMENT_PAID`). +- **Status Tracking**: Messages start in `SmsCreated` and transition to `SmsSent` or `SmsError` upon processing. + +### 3. Decoupled Data Synchronization +A separate scheduled job (Moqui `ServiceJob`) periodically polls for pending `SystemMessage` records. +- **Full Data Fetching**: The "Send" service associated with the message type fetches the full transitive entity tree using Moqui's standard `EntityDataDocument` logic. +- **Mapping**: The service maps Moqui's canonical JSON-like Map structure to the target system's (D365/NetSuite) API schema. +- **Guaranteed Delivery**: Provides built-in retries and error reporting via the `SystemMessage` UI. + +## Flow Diagram +```mermaid +sequenceDiagram + participant App as Moqui App + participant DB as Entity Engine (SQL DB) + participant Feed as DataFeed (Real-time) + participant Outbox as SystemMessage (Outbox) + participant Job as ServiceJob (Async) + participant ERP as D365 API + + App->>DB: Store Order (Status: OrderCancelled) + DB-->>Feed: Trigger Change Hook + Feed->>Outbox: Log Event (orderId, type: CANCEL) + Note over DB, Outbox: Committed in same transaction + + Job->>Outbox: Find pending messages + Outbox->>Job: return [orderId, type] + Job->>Job: Fetch full Order details + Job->>ERP: POST /SalesOrderLinesV3 (OData) + ERP-->>Job: 201 Created + Job->>Outbox: Update Status (SmsSent) +``` + +## Benefits +- **Reliability**: Uses JTA transaction synchronization to ensure database and integration logs remain atomic. +- **Decoupling**: The online transaction remains fast; the integration happens in the background. +- **Traceability**: All outbound messages are auditable via the `SystemMessage` interface. +- **Scalability**: Multiple processors can handle different `SystemMessageType` exports without interfering with each other. From 5e9b8703bfc729cc8ab616c68d0fc56ec8748db3 Mon Sep 17 00:00:00 2001 From: Gurveen Kaur Date: Tue, 24 Mar 2026 14:52:07 +0530 Subject: [PATCH 2/3] Correct DataFeed feedTypeEnumId to DTFDTP_RT_PUSH --- .../EVENT_DRIVEN_INTEGRATION_APPROACH.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md b/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md index 475de8bd..1aaf4414 100644 --- a/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md +++ b/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md @@ -10,13 +10,13 @@ Many traditional integrations in Moqui either rely on manual "sweeps" (polling) 3. **Difficult Monitoring**: No centralized "Integration Table" to track which events have been logged but not yet sent. ## The Solution: Moqui "Outbox" Pattern -We propose leveraging Moqui's native `DataFeed`, `DataDocument`, and `SystemMessage` infrastructure to implement an asynchronous **Outbox Pattern**. +We propose leveraging Moqui's native `DataFeed` and `DataDocument` infrastructure to implement an asynchronous **Outbox Pattern**. ### 1. Unified Event Detection (`DataFeed` & `DataDocument`) Instead of polling `OrderHeader` for status changes, we define a set of `DataDocument`s that capture the core event data. - **DataDocument**: Lightweight structure containing the `orderId` and `statusId`. -- **DataFeed**: A real-time (`DtfdptRtPush`) monitor that triggers on entity changes. It ensures transactional integrity by only firing after a successful DB commit. +- **DataFeed**: A real-time (`DTFDTP_RT_PUSH`) monitor that triggers on entity changes. It ensures transactional integrity by only firing after a successful DB commit. ### 2. Integration Event Log (`SystemMessage`) When a `DataFeed` triggers, it calls a lightweight logging service that generates a `SystemMessage`. This record acts as our persistent "Integration Table". From 3493ac251650224f0ce43de9668817bb2b34600d Mon Sep 17 00:00:00 2001 From: Gurveen Kaur Date: Tue, 24 Mar 2026 16:08:03 +0530 Subject: [PATCH 3/3] docs: finalize event-driven integration proposal for D365 --- .../EVENT_DRIVEN_INTEGRATION_APPROACH.md | 177 +++++++++++++----- 1 file changed, 131 insertions(+), 46 deletions(-) diff --git a/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md b/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md index 1aaf4414..828524c1 100644 --- a/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md +++ b/project-ideas/dynamics365-integration/EVENT_DRIVEN_INTEGRATION_APPROACH.md @@ -9,52 +9,137 @@ Many traditional integrations in Moqui either rely on manual "sweeps" (polling) 2. **Atomicity Issues**: If an integration service fails after a database change, the external system is out of sync. 3. **Difficult Monitoring**: No centralized "Integration Table" to track which events have been logged but not yet sent. -## The Solution: Moqui "Outbox" Pattern -We propose leveraging Moqui's native `DataFeed` and `DataDocument` infrastructure to implement an asynchronous **Outbox Pattern**. - -### 1. Unified Event Detection (`DataFeed` & `DataDocument`) -Instead of polling `OrderHeader` for status changes, we define a set of `DataDocument`s that capture the core event data. - -- **DataDocument**: Lightweight structure containing the `orderId` and `statusId`. -- **DataFeed**: A real-time (`DTFDTP_RT_PUSH`) monitor that triggers on entity changes. It ensures transactional integrity by only firing after a successful DB commit. - -### 2. Integration Event Log (`SystemMessage`) -When a `DataFeed` triggers, it calls a lightweight logging service that generates a `SystemMessage`. This record acts as our persistent "Integration Table". - -- **SystemMessageType**: Represents the event (e.g., `ORDER_CREATED`, `ORDER_CANCELLED`, `SHIPMENT_PAID`). -- **Status Tracking**: Messages start in `SmsCreated` and transition to `SmsSent` or `SmsError` upon processing. - -### 3. Decoupled Data Synchronization -A separate scheduled job (Moqui `ServiceJob`) periodically polls for pending `SystemMessage` records. -- **Full Data Fetching**: The "Send" service associated with the message type fetches the full transitive entity tree using Moqui's standard `EntityDataDocument` logic. -- **Mapping**: The service maps Moqui's canonical JSON-like Map structure to the target system's (D365/NetSuite) API schema. -- **Guaranteed Delivery**: Provides built-in retries and error reporting via the `SystemMessage` UI. - -## Flow Diagram -```mermaid -sequenceDiagram - participant App as Moqui App - participant DB as Entity Engine (SQL DB) - participant Feed as DataFeed (Real-time) - participant Outbox as SystemMessage (Outbox) - participant Job as ServiceJob (Async) - participant ERP as D365 API - - App->>DB: Store Order (Status: OrderCancelled) - DB-->>Feed: Trigger Change Hook - Feed->>Outbox: Log Event (orderId, type: CANCEL) - Note over DB, Outbox: Committed in same transaction +## The Solution: Moqui "Outbox" Pattern (Event-Driven) +We propose leveraging Moqui's native `DataFeed` and `DataDocument` infrastructure to implement an asynchronous **Outbox Pattern**, driven by status history events. + +### 1. Immutable Event Detection (`OrderStatus` / `ShipmentStatus`) +Instead of polling the header tables (where fields can change multiple times), we monitor the **Status History** entities. +- **Mechanism**: Every status change creates a *new* record in the history table. By making this the primary entity of our `DataFeed`, we ensure that the integration triggers exactly **once** per status transition. +- **Transactional Integrity**: The `DataFeed` only fires after a successful database commit, ensuring data consistency. + +## 2. Outbox Implementation Options +Once a `DataFeed` triggers, the event is logged to a persistent store: + +### Option A: Custom Integration Table (Manual Outbox) +- **Flow**: `DataFeed` -> `log#IntegrationEvent` -> `D365IntegrationEvent`. +- **Pros**: Highly specialized for D365 specifics (like custom batch IDs). +- **Cons**: Requires manual development of status management and retry logic. + +### Option B: SystemMessage (Framework Native) +- **Flow**: `DataFeed` -> `log#IntegrationEvent` -> `SystemMessage`. +- **Pros**: Built-in error capturing (full stack traces), retry limits, and a professional Monitoring UI. +- **Cons**: Fixed status types. + +## 3. Comparative Tradeoff Analysis + +| Feature | Custom Integration Table | **SystemMessage (Framework)** | +| :--- | :--- | :--- | +| **Development Effort** | **High** (Build status/retry logic from scratch). | **Low** (Reuse existing framework code). | +| **Error Logging** | Manual (Must store API error text). | **Automatic** (Captures full Java stack traces). | +| **Monitoring UI** | Requires building a custom Screen. | **Standard Screen** (`SystemMessage`). | +| **Reliability** | Depends on custom service quality. | **Proven** (Used globally in Moqui integrations). | + +## Event Mapping Table +The following table defines the proposed mappings using the **Event-First** approach. + +| Event | Primary Entity (History) | Related Entity | Trigger Condition | +| :--- | :--- | :--- | :--- | +| **Order Created** | `OrderStatus` | `OrderHeader` | `statusId == 'ORDER_CREATED'` | +| **Order Cancelled** | `OrderStatus` | `OrderHeader` | `statusId == 'ORDER_CANCELLED'` | +| **Order Fulfillment** | `ShipmentStatus` | `Shipment` | `statusId == 'SHIPMENT_SHIPPED'` | +| **Return Completed** | `ReturnStatus` | `ReturnHeader` | New `RETURN_COMPLETED` record | + +## Implementation Examples (XML Definition) + +### 1. Sales Order Created +```xml + + + + + - Job->>Outbox: Find pending messages - Outbox->>Job: return [orderId, type] - Job->>Job: Fetch full Order details - Job->>ERP: POST /SalesOrderLinesV3 (OData) - ERP-->>Job: 201 Created - Job->>Outbox: Update Status (SmsSent) + + + + + +``` + +### 2. Sales Order Cancelled +```xml + + + + + + + + + + +``` + +### 3. Sales Order Fulfillment (Shipped) +```xml + + + + + + + + + +``` + +### 4. Sales Order Return Completed +```xml + + + + + + + ``` -## Benefits -- **Reliability**: Uses JTA transaction synchronization to ensure database and integration logs remain atomic. -- **Decoupling**: The online transaction remains fast; the integration happens in the background. -- **Traceability**: All outbound messages are auditable via the `SystemMessage` interface. -- **Scalability**: Multiple processors can handle different `SystemMessageType` exports without interfering with each other. + + ## Data Feed Configuration + Multiple `DataDocument`s can be attached to the same `DataFeed` to funnel all related events into a single dispatcher service. + + ```xml + + + + + + + + + + + + + + + + ``` + + ## Benefits +- **Atomicity**: The `DataFeed` mechanism ensures the outbox log is created only if the business transaction succeeds. +- **Deduplication**: History-based triggering naturally prevents redundant events for a single transition. +- **Traceability**: All outbound messages are auditable via the framework's native tools. +- **Scalability**: Decoupling allows for high-throughput background processing without impact on customer-facing performance.