Technical documentation for new Shopify product sync workflow

project-ideas/product-master/new-design/product-sync-outline.md

@@ -0,0 +1,17 @@
+# Shopify Product Sync Design Outline
+
+This document describes the design for the product synchronization between Shopify and the OMS. The sync process is divided into clear stages to ensure reliability and handle API constraints.
+
+### The Stages of Sync:
+
+1.  **Queuing the Request**: We create a system message to fetch data for products that were recently updated. This message has the filters needed for the Shopify GraphQL query.
+2.  **Sending to Shopify**: We send the message to Shopify to start a "Bulk Operation." We only do this if Shopify is not already busy with another bulk operation.
+3.  **Checking Completion**: We keep checking (polling) until the bulk operation is finished at Shopify. Once it's done, we get the result file link.
+4.  **Data Preparation**: We download the JSONL file. Then we convert this data into a nested JSON list and send it to the MDM (OMS) layer.
+5.  **Diff Computation**: The system looks at the new incoming data and compares it with the "Last Processed Data." This tells us exactly what has changed (Price, Title, Tags, etc.).
+6.  **Updating Database**: We apply only the changes (the "diffs") to the database. This keeps the update very fast and clean.
+7.  **Saving History**: Finally, we save the new data as a "history" snapshot. This will be used as the base for comparison the next time the sync runs.
+
+---
+
+This outline shows the complete workflow of a product sync, from the moment a change is detected in Shopify to the final update in our database.

project-ideas/product-master/new-design/step-1-queuing-bulk-query.md

@@ -0,0 +1,51 @@
+# Step 1: Queuing the Bulk Query Message
+
+In this step, we just create a system message entry to track what data we need from Shopify. We don't call the Shopify API yet because we only want to plan the sync here.
+
+### Things involved
+1.  **Service**: `co.hotwax.shopify.system.ShopifySystemMessageServices.queue#BulkQuerySystemMessage`. This is the core service that prepares the request.
+2.  **System Message Type**: `BulkProductAndVariantsByIdQuery`.
+
+    Here is the data setup for this type:
+    ```json
+    {
+        "systemMessageTypeId": "BulkProductAndVariantsByIdQuery",
+        "parentTypeId": "ShopifyBulkQuery",
+        "sendServiceName": "co.hotwax.shopify.system.ShopifySystemMessageServices.send#BulkQuerySystemMessage",
+        "consumeServiceName": "co.hotwax.shopify.system.ShopifySystemMessageServices.consume#ProductVariantUpdates",
+        "sendPath": "component://shopify-connector/template/graphQL/BulkProductAndVariantsByIdQuery.ftl",
+        "_entity": "moqui.service.message.SystemMessageType"
+    }
+    ```
+
+3.  **Data Parameters**:
+    - `systemMessageTypeId`: `BulkProductAndVariantsByIdQuery`
+    - `systemMessageRemoteId`: The ID for the Shopify shop.
+    - `fromDateBuffer`: A value (usually in minutes) used to avoid missing data.
+
+---
+
+### How the flow works:
+
+1.  **Job Execution**: A scheduled job runs and calls the service. It passes the `systemMessageTypeId` and `systemMessageRemoteId`.
+2.  **Date Computation**: 
+    - The service first looks for the last successful message (`statusId: SmsgConfirmed`) to get the `processedDate`. 
+    - It uses this date as the starting point. It also subtracts the `fromDateBuffer` minutes and converts the final time to **UTC format** for Shopify.
+    - If `filterQuery` or `thruDate` are provided in the job, the service handles them too.
+3.  **Preparing Message Text**: All these parameters (dates, filters) are converted into a JSON string. This JSON is saved in the `messageText` field of the new system message.
+4.  **Creating the Entry**: The service calls `org.moqui.impl.SystemMessageServices.queue#SystemMessage` to create the record:
+    - **statusId**: `SmsgProduced`
+    - **isOutgoing**: `Y`
+    - **sendNow**: `false` (This is very important so the sync doesn't start immediately).
+
+By keeping `sendNow` as `false`, we just put the "request" in the queue. The next job will pick it up only when no other bulk operation is running on Shopify.
+
+---
+
+### Service Call Chain:
+
+1.  **Scheduled Job**: `queue_BulkQuerySystemMessage_BulkProductAndVariantsByIdQuery`
+2.  **mantle-shopify-connector**: `co.hotwax.shopify.system.ShopifySystemMessageServices.queue#BulkQuerySystemMessage`
+    -   **Action**: Computes dates and prepares JSON parameters.
+3.  **moqui-framework**: `org.moqui.impl.SystemMessageServices.queue#SystemMessage`
+    -   **Action**: Saves the message as `SmsgProduced` with `sendNow=false`.

project-ideas/product-master/new-design/step-2-sending-bulk-query.md

@@ -0,0 +1,52 @@
+# Step 2: Sending the Bulk Query Message
+
+This step handles the actual Shopify API call. We built it this way specifically to deal with **Shopify’s limit**: only one bulk operation can run at a time per shop.
+
+### Things involved
+1.  **Scheduler**: `co.hotwax.shopify.system.ShopifySystemMessageServices.send#ProducedBulkOperationSystemMessage`. This is a scheduled service that looks for queued work.
+2.  **Parent Group**: `ShopifyBulkQuery`. This is the `parentTypeId` used to group all bulk requests (like Product, Inventory, and Order queries). 
+3.  **The Sender**: `co.hotwax.shopify.system.ShopifySystemMessageServices.send#BulkQuerySystemMessage`. This is the service that actually builds the Shopify request.
+
+---
+
+### Detailed Technical Flow:
+
+#### 1. The Locking Check
+When the scheduled job runs, it passes a **`parentSystemMessageTypeId`** (ShopifyBulkQuery). 
+The service then does an `entity-find` on the `SystemMessage` table. 
+*   **Search**: It looks for any message where `statusId == 'SmsgSent'` and the `parentTypeId` matches.
+*   **The Constraint**: If it finds even one record, it logs a message: *"Aborting, ShopifyBulkQuery Operation already in progress."* and stops. This prevents sending another request while Shopify is still busy.
+
+#### 2. Selecting the next Message
+If no active (`SmsgSent`) message is found:
+*   **Filtering**: It looks for messages in `SmsgProduced` status with the same `parentTypeId`.
+*   **Ordering**: It picks the message with the oldest **`initDate`** (First-In, First-Out).
+*   **Retry Logic**: It checks the **`failCount`**. If the count is less than the **`retryLimit`** (default is 3), it moves to the send step. If it has failed too many times, it sets the status to `SmsgError`.
+
+#### 3. Calling Shopify API
+The manager calls the **Sender service**. This service does the following:
+*   **Credentials**: It uses the `systemMessageRemoteId` from the message to get the correct Shopify API keys.
+*   **Building the Query**: It reads the JSON dates and filters from the `messageText`. It then uses the **`sendPath`** (which points to an `.ftl` file) to build the complete GraphQL mutation.
+*   **HTTP POST**: It calls `co.hotwax.shopify.common.ShopifyHelperServices.send#ShopifyGraphqlRequest` to send a POST request to Shopify.
+
+#### 4. Storing the Shopify ID
+Once Shopify confirms the mutation started:
+- The system gets the `id` of the bulk operation from Shopify's response.
+- It saves this ID in the **`remoteMessageId`** field of our system message.
+- It updates the status to **`SmsgSent`**.
+
+Now the system is "Locked." Any other bulk query job that runs now will see this `SmsgSent` message and wait until it is finished.
+
+---
+
+### Service Call Chain:
+
+1.  **Scheduled Job**: `send_ProducedBulkOperationSystemMessage_ShopifyBulkQuery`
+2.  **mantle-shopify-connector**: `co.hotwax.shopify.system.ShopifySystemMessageServices.send#ProducedBulkOperationSystemMessage`
+    -   **Action**: Checks for active `SmsgSent` messages to handle the "Busy" lock.
+3.  **moqui-framework**: `org.moqui.impl.SystemMessageServices.send#ProducedSystemMessage`
+    -   **Action**: Standard framework service to trigger the sending process.
+4.  **mantle-shopify-connector**: `co.hotwax.shopify.system.ShopifySystemMessageServices.send#BulkQuerySystemMessage`
+    -   **Action**: Expands the FTL template and prepares the GraphQL mutation.
+5.  **mantle-shopify-connector**: `co.hotwax.shopify.common.ShopifyHelperServices.send#ShopifyGraphqlRequest`
+    -   **Action**: Sends the request to Shopify and updates status to `SmsgSent`.

project-ideas/product-master/new-design/step-3-confirming-bulk-operation.md

@@ -0,0 +1,62 @@
+# Step 3: Confirming the Bulk Operation
+
+In this step, we get the final confirmation from Shopify. We use two parallel methods: **Polling** and **Webhooks**. They both help to reach the same result by calling the same processing service.
+
+### Things involved
+1.  **Poller Service**: `co.hotwax.shopify.system.ShopifySystemMessageServices.poll#BulkOperationResult`.
+2.  **Processor Service**: `co.hotwax.shopify.system.ShopifySystemMessageServices.process#BulkOperationResult`. This handles the final result logic for both methods.
+3.  **Webhook Type**: `BulkOperationsFinish`. This listens for the Shopify event when a bulk operation finishes.
+4.  **Endpoint**: `/rest/s1/shopify/webhook/payload` (mapped to `co.hotwax.shopify.webhook.ShopifyWebhookServices.receive#WebhookPayload`).
+5.  **Status Mapping**: 
+    - `completed` -> **`SmsgConfirmed`**
+    - `canceled` -> **`SmsgCancelled`**
+    - `failed` or `expired` -> **`SmsgError`**
+
+---
+
+### Two Ways to Confirm
+
+#### Method 1: Polling (Direct Check)
+The scheduled job `poll_BulkOperationResult_ShopifyBulkQuery` runs this path.
+*   **Search**: It looks for any message with status **`SmsgSent`** in the `ShopifyBulkQuery` parent group.
+*   **Checking Shopify**: It takes the **`remoteMessageId`** and calls Shopify's API via `co.hotwax.shopify.graphQL.ShopifyBulkImportServices.get#BulkOperationResult`.
+*   **Finalizing**: Once it sees the status is `COMPLETED`, it calls the **Processor Service**.
+
+#### Method 2: Webhooks (Real-time Notification)
+This method receives a direct "Finished" update from Shopify.
+*   **Reception**: Shopify sends a payload to our system. This creates an incoming message of type `BulkOperationsFinish`.
+*   **Finding the Request**: The consumer service `co.hotwax.shopify.system.ShopifySystemMessageServices.consume#BulkOperationsFinishWebhookPayload` reads the Shopify ID and finds our matching **Outgoing** sync message.
+*   **Finalizing**: It then calls the same **Processor Service** (`process#BulkOperationResult`).
+
+---
+
+### The Final Process (Common Logic)
+The `co.hotwax.shopify.system.ShopifySystemMessageServices.process#BulkOperationResult` service manages the final work:
+1.  **Error Handling**: If Shopify returns an error, the service creates a record in the **`SystemMessageError`** table to log exactly why it failed.
+2.  **Release the Sync Queue**: It updates the outgoing message status to **`SmsgConfirmed`**. This releases the "lock" so Step 2 can send the next sync job in the queue.
+3.  **Create the Results Message**:
+    - It creates a **New Incoming System Message** (using `org.moqui.impl.SystemMessageServices.receive#IncomingSystemMessage`).
+    - It saves the Shopify **result file URL** in the `messageText`.
+    - It populates the **`parentMessageId`** with our original **`systemMessageId`**. This is how we keep the vertical link for full traceability back to the original request.
+    - This link allows us to know exactly which bulk query produced which result file.
+
+---
+
+### Key Technical Notes:
+*   **Polling Limit**: The poller service only checks **one** message at a time (`limit="1"`). This keeps the process focused if many shops are running at once.
+*   **Traceability**: Using **`parentMessageId`** is the standard way we link the incoming result message back to the outgoing request.
+
+---
+
+### Service Call Chain:
+
+#### For Polling:
+1.  **Scheduled Job** $\rightarrow$ `co.hotwax.shopify.system.ShopifySystemMessageServices.poll#BulkOperationResult`
+2.  `poll#BulkOperationResult` $\rightarrow$ `co.hotwax.shopify.system.ShopifySystemMessageServices.process#BulkOperationResult`
+3.  `process#BulkOperationResult` $\rightarrow$ `org.moqui.impl.SystemMessageServices.receive#IncomingSystemMessage`
+
+#### For Webhook:
+1.  Shopify sends `bulk_operations/finish` notification.
+2.  **`co.hotwax.shopify.webhook.ShopifyWebhookServices.receive#WebhookPayload`** creates an incoming message.
+3.  `co.hotwax.shopify.system.ShopifySystemMessageServices.consume#BulkOperationsFinishWebhookPayload` $\rightarrow$ `co.hotwax.shopify.system.ShopifySystemMessageServices.process#BulkOperationResult`
+4.  `process#BulkOperationResult` handles final status and result link logic.

project-ideas/product-master/new-design/step-4-data-preparation.md

@@ -0,0 +1,81 @@
+# Step 4: Data Preparation (JSONL to Nested JSON)
+
+This step handles the transformation of the raw Shopify bulk result (JSONL) into a hierarchical JSON format that the OMS can process.
+
+## Control Flow (How we reach this step)
+Control reaches this step immediately after the Shopify Bulk Operation is confirmed as `completed`. The sequence is as follows:
+
+1.  **Confirmation (Step 3)**: The `process#BulkOperationResult` service identifies the operation is complete and calls `receive#IncomingSystemMessage`.
+2.  **Direct Trigger**: Since the `BulkProductAndVariantsByIdQuery` message type is configured with **`consume#ProductVariantUpdates`** as its consume service, the System Message framework triggers this step directly.
+3.  **Data Preparation**: This service (`consume#ProductVariantUpdates`) then reads the download URL, streams the JSONL, converts it to nested JSON, and uploads it to the MDM.
+
+## 1. Input: Raw JSONL Data
+Shopify bulk results are returned as a "flat" JSONL file where child entities (Variants, Metafields) follow their parents and reference them via a `__parentId`.
+
+**Example (Simplified JSONL):**
+```json
+{"id":"gid://shopify/Product/9028462969133","handle":"doria-denim","title":"Doria - Denim"}
+{"id":"gid://shopify/ProductVariant/47646836457773","sku":"5138980_denim_50","__parentId":"gid://shopify/Product/9028462969133"}
+{"id":"gid://shopify/Metafield/31849382215981","key":"show_width_code","value":"G","__parentId":"gid://shopify/Product/9028462969133"}
+```
+
+## 2. Transformation Logic
+The `consume#ProductVariantUpdates` service parses this file line-by-line and groups child entities under their respective parent objects based on the `__parentId`.
+
+- **Root Object**: Any line without a `__parentId` is treated as a new Product.
+- **Child Object**: Any line with a `__parentId` is added to a list inside the current Product object. The list name is derived from the object type in the Shopify GID (e.g., `ProductVariant`, `Metafield`).
+
+## 3. Output: Nested JSON
+The transformation produces a standard JSON array where each product contains its own nested variants and metadata.
+
+**Example (Transformed JSON):**
+```json
+[
+  {
+    "virtualProduct": {
+      "id": "gid://shopify/Product/9028462969133",
+      "handle": "doria-denim",
+      "title": "Doria - Denim",
+      "ProductVariant": [
+        {
+          "id": "gid://shopify/ProductVariant/47646836457773",
+          "sku": "5138980_denim_50"
+        }
+      ],
+      "Metafield": [
+        {
+          "id": "gid://shopify/Metafield/31849382215981",
+          "key": "show_width_code",
+          "value": "G"
+        }
+      ]
+    }
+  }
+]
+```
+
+## 4. Upload to MDM
+Once the file is converted, it is uploaded to the **Data Manager (MDM)** for processing.
+
+- **Service**: `co.hotwax.util.UtilityServices.upload#DataManagerFile`
+- **Config ID**: `SYNC_SHOPIFY_PRODUCT`
+- **DataManager Configuration**:
+  ```json
+  {
+      "importServiceName": "co.hotwax.sob.product.ProductServices.sync#ShopifyProduct",
+      "executionModeId": "DMC_QUEUE",
+      "configId": "SYNC_SHOPIFY_PRODUCT"
+  }
+  ```
+- **Traceability**: The `systemMessageId` is passed in the `parameters` map to maintain a link between this data batch and the original sync request.
+
+## Service Call Chain
+1. `co.hotwax.shopify.system.ShopifySystemMessageServices.process#BulkOperationResult` (Step 3: detects completion)
+2. `org.moqui.impl.SystemMessageServices.receive#IncomingSystemMessage` (Creates incoming message and triggers consume)
+3. `co.hotwax.shopify.system.ShopifySystemMessageServices.consume#ProductVariantUpdates` (This step: conversion & upload)
+4. `co.hotwax.util.UtilityServices.upload#DataManagerFile` (Upload to MDM)
+
+## Technical Notes
+- **Streaming**: The data is processed line-by-line using a `JsonGenerator` to handle potentially large files without exhausting memory.
+- **Order Dependency**: The parser expects parents to precede their children in the Shopify file.
+- **Traceability**: Uses `parentMessageId` (from previous steps) and passes `systemMessageId` to the Data Manager to ensure full end-to-end logging.

project-ideas/product-master/new-design/step-5-diff-computation.md

@@ -0,0 +1,60 @@
+# Step 5: Diff Computation (Change Detection)
+
+Once the nested JSON is received by the Data Manager, the core logic in `syncShopifyProduct.groovy` begins the process of identifying changes. Instead of blindly overwriting the database, it uses a **Baseline Comparison** strategy to perform selective updates.
+
+## 1. History Lookup (The Baseline)
+For every Product or Variant in the incoming JSON, the service fetches the last synchronized state from the history table.
+
+- **Entity**: `co.hotwax.product.ProductUpdateHistory`
+- **Lookup Key**: `productId` (Shopify GID) and `shopId`.
+- **Purpose**: This record stores hashes and data snapshots from the *previous* sync, serving as the benchmark for change detection.
+
+## 2. Hashing Logic
+To detect changes efficiently, the service groups product data into "buckets" and computes a **SHA-256 Hash** for each.
+
+| Data Bucket | Fields Included in Hash |
+| :--- | :--- |
+| **Core Details** | Title, Handle, Vendor, Featured Image URL, shipping/giftCard flags. |
+| **Tags** | A sorted list of Shopify tags. |
+| **Features** | A sorted list of product options (Position, Name, Value). |
+| **Metafields** | A list of normalized metafields (Namespace, Key, Value). |
+| **Assocs** | A sorted list of variant IDs associated with the product. |
+
+**Why Hashing?** Comparing a single 64-character hash is significantly faster than comparing dozens of individual fields or large JSON blobs.
+
+## 3. Delta Identification
+If an incoming hash does not match the stored hash, the service performs a deep comparison to identify the exact delta.
+
+### List Comparison (Tags, Features, Metafields)
+For lists, it identifies exactly which items are new and which have been removed:
+- **`added`**: Items present in incoming data but missing in history.
+- **`removed`**: Items present in history but missing in incoming data.
+
+### Field Comparison (Price, Title, etc.)
+For simple fields like Price or weight, it performs a standard value comparison (e.g., using `BigDecimal` for price accuracy).
+
+## 4. Output: The `differenceMap`
+The result of this step is a specialized JSON object called the **`differenceMap`**. This map is the "instruction set" for the next step (Ingestion).
+
+**Example `differenceMap`:**
+```json
+{
+  "title": "Doria Denim - Updated",
+  "tags": {
+    "added": ["New Arrival", "Summer 2024"],
+    "removed": ["Sale"]
+  },
+  "features": {
+    "added": [{"name": "Width", "value": "Wide"}],
+    "removed": []
+  }
+}
+```
+
+## Service Call Chain
+1. `co.hotwax.sob.product.ProductServices.sync#ShopifyProduct` (Main Entry)
+2. `compareVirtualProduct` / `compareVariantProduct` (Inner Logic)
+3. `DigestUtils.sha256Hex` (Hashing Utility)
+
+## Traceability
+The `systemMessageId` is carried forward and associated with the logic to ensure that every diff can be traced back to the specific Shopify sync request.