diff --git a/project-ideas/product-master/diagrams/current-flow/current-sync-detailed-sequence.png b/project-ideas/product-master/diagrams/current-flow/current-sync-detailed-sequence.png new file mode 100644 index 00000000..80fa8b74 Binary files /dev/null and b/project-ideas/product-master/diagrams/current-flow/current-sync-detailed-sequence.png differ diff --git a/project-ideas/product-master/diagrams/current-flow/current-transformation-ingestion-architecture.png b/project-ideas/product-master/diagrams/current-flow/current-transformation-ingestion-architecture.png new file mode 100644 index 00000000..f3380d09 Binary files /dev/null and b/project-ideas/product-master/diagrams/current-flow/current-transformation-ingestion-architecture.png differ diff --git a/project-ideas/product-master/new-design/product-sync-outline.md b/project-ideas/product-master/new-design/product-sync-outline.md new file mode 100644 index 00000000..736a64d8 --- /dev/null +++ b/project-ideas/product-master/new-design/product-sync-outline.md @@ -0,0 +1,17 @@ +# Shopify Product Sync Design Outline + +This document describes the design for the product synchronization between Shopify and the OMS. The sync process is divided into clear stages to ensure reliability and handle API constraints. + +### The Stages of Sync: + +1. **Queuing the Request**: We create a system message to fetch data for products that were recently updated. This message has the filters needed for the Shopify GraphQL query. +2. **Sending to Shopify**: We send the message to Shopify to start a "Bulk Operation." We only do this if Shopify is not already busy with another bulk operation. +3. **Checking Completion**: We keep checking (polling) until the bulk operation is finished at Shopify. Once it's done, we get the result file link. +4. **Data Preparation**: We download the JSONL file. Then we convert this data into a nested JSON list and send it to the MDM (OMS) layer. +5. **Diff Computation**: The system looks at the new incoming data and compares it with the "Last Processed Data." This tells us exactly what has changed (Price, Title, Tags, etc.). +6. **Updating Database**: We apply only the changes (the "diffs") to the database. This keeps the update very fast and clean. +7. **Saving History**: Finally, we save the new data as a "history" snapshot. This will be used as the base for comparison the next time the sync runs. + +--- + +This outline shows the complete workflow of a product sync, from the moment a change is detected in Shopify to the final update in our database. diff --git a/project-ideas/product-master/new-design/step-1-queuing-bulk-query.md b/project-ideas/product-master/new-design/step-1-queuing-bulk-query.md new file mode 100644 index 00000000..fc4422c5 --- /dev/null +++ b/project-ideas/product-master/new-design/step-1-queuing-bulk-query.md @@ -0,0 +1,51 @@ +# Step 1: Queuing the Bulk Query Message + +In this step, we just create a system message entry to track what data we need from Shopify. We don't call the Shopify API yet because we only want to plan the sync here. + +### Things involved +1. **Service**: `co.hotwax.shopify.system.ShopifySystemMessageServices.queue#BulkQuerySystemMessage`. This is the core service that prepares the request. +2. **System Message Type**: `BulkProductAndVariantsByIdQuery`. + + Here is the data setup for this type: + ```json + { + "systemMessageTypeId": "BulkProductAndVariantsByIdQuery", + "parentTypeId": "ShopifyBulkQuery", + "sendServiceName": "co.hotwax.shopify.system.ShopifySystemMessageServices.send#BulkQuerySystemMessage", + "consumeServiceName": "co.hotwax.shopify.system.ShopifySystemMessageServices.consume#ProductVariantUpdates", + "sendPath": "component://shopify-connector/template/graphQL/BulkProductAndVariantsByIdQuery.ftl", + "_entity": "moqui.service.message.SystemMessageType" + } + ``` + +3. **Data Parameters**: + - `systemMessageTypeId`: `BulkProductAndVariantsByIdQuery` + - `systemMessageRemoteId`: The ID for the Shopify shop. + - `fromDateBuffer`: A value (usually in minutes) used to avoid missing data. + +--- + +### How the flow works: + +1. **Job Execution**: A scheduled job runs and calls the service. It passes the `systemMessageTypeId` and `systemMessageRemoteId`. +2. **Date Computation**: + - The service first looks for the last successful message (`statusId: SmsgConfirmed`) to get the `processedDate`. + - It uses this date as the starting point. It also subtracts the `fromDateBuffer` minutes and converts the final time to **UTC format** for Shopify. + - If `filterQuery` or `thruDate` are provided in the job, the service handles them too. +3. **Preparing Message Text**: All these parameters (dates, filters) are converted into a JSON string. This JSON is saved in the `messageText` field of the new system message. +4. **Creating the Entry**: The service calls `org.moqui.impl.SystemMessageServices.queue#SystemMessage` to create the record: + - **statusId**: `SmsgProduced` + - **isOutgoing**: `Y` + - **sendNow**: `false` (This is very important so the sync doesn't start immediately). + +By keeping `sendNow` as `false`, we just put the "request" in the queue. The next job will pick it up only when no other bulk operation is running on Shopify. + +--- + +### Service Call Chain: + +1. **Scheduled Job**: `queue_BulkQuerySystemMessage_BulkProductAndVariantsByIdQuery` +2. **mantle-shopify-connector**: `co.hotwax.shopify.system.ShopifySystemMessageServices.queue#BulkQuerySystemMessage` + - **Action**: Computes dates and prepares JSON parameters. +3. **moqui-framework**: `org.moqui.impl.SystemMessageServices.queue#SystemMessage` + - **Action**: Saves the message as `SmsgProduced` with `sendNow=false`. diff --git a/project-ideas/product-master/new-design/step-2-sending-bulk-query.md b/project-ideas/product-master/new-design/step-2-sending-bulk-query.md new file mode 100644 index 00000000..facc1313 --- /dev/null +++ b/project-ideas/product-master/new-design/step-2-sending-bulk-query.md @@ -0,0 +1,52 @@ +# Step 2: Sending the Bulk Query Message + +This step handles the actual Shopify API call. We built it this way specifically to deal with **Shopify’s limit**: only one bulk operation can run at a time per shop. + +### Things involved +1. **Scheduler**: `co.hotwax.shopify.system.ShopifySystemMessageServices.send#ProducedBulkOperationSystemMessage`. This is a scheduled service that looks for queued work. +2. **Parent Group**: `ShopifyBulkQuery`. This is the `parentTypeId` used to group all bulk requests (like Product, Inventory, and Order queries). +3. **The Sender**: `co.hotwax.shopify.system.ShopifySystemMessageServices.send#BulkQuerySystemMessage`. This is the service that actually builds the Shopify request. + +--- + +### Detailed Technical Flow: + +#### 1. The Locking Check +When the scheduled job runs, it passes a **`parentSystemMessageTypeId`** (ShopifyBulkQuery). +The service then does an `entity-find` on the `SystemMessage` table. +* **Search**: It looks for any message where `statusId == 'SmsgSent'` and the `parentTypeId` matches. +* **The Constraint**: If it finds even one record, it logs a message: *"Aborting, ShopifyBulkQuery Operation already in progress."* and stops. This prevents sending another request while Shopify is still busy. + +#### 2. Selecting the next Message +If no active (`SmsgSent`) message is found: +* **Filtering**: It looks for messages in `SmsgProduced` status with the same `parentTypeId`. +* **Ordering**: It picks the message with the oldest **`initDate`** (First-In, First-Out). +* **Retry Logic**: It checks the **`failCount`**. If the count is less than the **`retryLimit`** (default is 3), it moves to the send step. If it has failed too many times, it sets the status to `SmsgError`. + +#### 3. Calling Shopify API +The manager calls the **Sender service**. This service does the following: +* **Credentials**: It uses the `systemMessageRemoteId` from the message to get the correct Shopify API keys. +* **Building the Query**: It reads the JSON dates and filters from the `messageText`. It then uses the **`sendPath`** (which points to an `.ftl` file) to build the complete GraphQL mutation. +* **HTTP POST**: It calls `co.hotwax.shopify.common.ShopifyHelperServices.send#ShopifyGraphqlRequest` to send a POST request to Shopify. + +#### 4. Storing the Shopify ID +Once Shopify confirms the mutation started: +- The system gets the `id` of the bulk operation from Shopify's response. +- It saves this ID in the **`remoteMessageId`** field of our system message. +- It updates the status to **`SmsgSent`**. + +Now the system is "Locked." Any other bulk query job that runs now will see this `SmsgSent` message and wait until it is finished. + +--- + +### Service Call Chain: + +1. **Scheduled Job**: `send_ProducedBulkOperationSystemMessage_ShopifyBulkQuery` +2. **mantle-shopify-connector**: `co.hotwax.shopify.system.ShopifySystemMessageServices.send#ProducedBulkOperationSystemMessage` + - **Action**: Checks for active `SmsgSent` messages to handle the "Busy" lock. +3. **moqui-framework**: `org.moqui.impl.SystemMessageServices.send#ProducedSystemMessage` + - **Action**: Standard framework service to trigger the sending process. +4. **mantle-shopify-connector**: `co.hotwax.shopify.system.ShopifySystemMessageServices.send#BulkQuerySystemMessage` + - **Action**: Expands the FTL template and prepares the GraphQL mutation. +5. **mantle-shopify-connector**: `co.hotwax.shopify.common.ShopifyHelperServices.send#ShopifyGraphqlRequest` + - **Action**: Sends the request to Shopify and updates status to `SmsgSent`. diff --git a/project-ideas/product-master/new-design/step-3-confirming-bulk-operation.md b/project-ideas/product-master/new-design/step-3-confirming-bulk-operation.md new file mode 100644 index 00000000..637d81e7 --- /dev/null +++ b/project-ideas/product-master/new-design/step-3-confirming-bulk-operation.md @@ -0,0 +1,62 @@ +# Step 3: Confirming the Bulk Operation + +In this step, we get the final confirmation from Shopify. We use two parallel methods: **Polling** and **Webhooks**. They both help to reach the same result by calling the same processing service. + +### Things involved +1. **Poller Service**: `co.hotwax.shopify.system.ShopifySystemMessageServices.poll#BulkOperationResult`. +2. **Processor Service**: `co.hotwax.shopify.system.ShopifySystemMessageServices.process#BulkOperationResult`. This handles the final result logic for both methods. +3. **Webhook Type**: `BulkOperationsFinish`. This listens for the Shopify event when a bulk operation finishes. +4. **Endpoint**: `/rest/s1/shopify/webhook/payload` (mapped to `co.hotwax.shopify.webhook.ShopifyWebhookServices.receive#WebhookPayload`). +5. **Status Mapping**: + - `completed` -> **`SmsgConfirmed`** + - `canceled` -> **`SmsgCancelled`** + - `failed` or `expired` -> **`SmsgError`** + +--- + +### Two Ways to Confirm + +#### Method 1: Polling (Direct Check) +The scheduled job `poll_BulkOperationResult_ShopifyBulkQuery` runs this path. +* **Search**: It looks for any message with status **`SmsgSent`** in the `ShopifyBulkQuery` parent group. +* **Checking Shopify**: It takes the **`remoteMessageId`** and calls Shopify's API via `co.hotwax.shopify.graphQL.ShopifyBulkImportServices.get#BulkOperationResult`. +* **Finalizing**: Once it sees the status is `COMPLETED`, it calls the **Processor Service**. + +#### Method 2: Webhooks (Real-time Notification) +This method receives a direct "Finished" update from Shopify. +* **Reception**: Shopify sends a payload to our system. This creates an incoming message of type `BulkOperationsFinish`. +* **Finding the Request**: The consumer service `co.hotwax.shopify.system.ShopifySystemMessageServices.consume#BulkOperationsFinishWebhookPayload` reads the Shopify ID and finds our matching **Outgoing** sync message. +* **Finalizing**: It then calls the same **Processor Service** (`process#BulkOperationResult`). + +--- + +### The Final Process (Common Logic) +The `co.hotwax.shopify.system.ShopifySystemMessageServices.process#BulkOperationResult` service manages the final work: +1. **Error Handling**: If Shopify returns an error, the service creates a record in the **`SystemMessageError`** table to log exactly why it failed. +2. **Release the Sync Queue**: It updates the outgoing message status to **`SmsgConfirmed`**. This releases the "lock" so Step 2 can send the next sync job in the queue. +3. **Create the Results Message**: + - It creates a **New Incoming System Message** (using `org.moqui.impl.SystemMessageServices.receive#IncomingSystemMessage`). + - It saves the Shopify **result file URL** in the `messageText`. + - It populates the **`parentMessageId`** with our original **`systemMessageId`**. This is how we keep the vertical link for full traceability back to the original request. + - This link allows us to know exactly which bulk query produced which result file. + +--- + +### Key Technical Notes: +* **Polling Limit**: The poller service only checks **one** message at a time (`limit="1"`). This keeps the process focused if many shops are running at once. +* **Traceability**: Using **`parentMessageId`** is the standard way we link the incoming result message back to the outgoing request. + +--- + +### Service Call Chain: + +#### For Polling: +1. **Scheduled Job** $\rightarrow$ `co.hotwax.shopify.system.ShopifySystemMessageServices.poll#BulkOperationResult` +2. `poll#BulkOperationResult` $\rightarrow$ `co.hotwax.shopify.system.ShopifySystemMessageServices.process#BulkOperationResult` +3. `process#BulkOperationResult` $\rightarrow$ `org.moqui.impl.SystemMessageServices.receive#IncomingSystemMessage` + +#### For Webhook: +1. Shopify sends `bulk_operations/finish` notification. +2. **`co.hotwax.shopify.webhook.ShopifyWebhookServices.receive#WebhookPayload`** creates an incoming message. +3. `co.hotwax.shopify.system.ShopifySystemMessageServices.consume#BulkOperationsFinishWebhookPayload` $\rightarrow$ `co.hotwax.shopify.system.ShopifySystemMessageServices.process#BulkOperationResult` +4. `process#BulkOperationResult` handles final status and result link logic. diff --git a/project-ideas/product-master/new-design/step-4-data-preparation.md b/project-ideas/product-master/new-design/step-4-data-preparation.md new file mode 100644 index 00000000..49988d5e --- /dev/null +++ b/project-ideas/product-master/new-design/step-4-data-preparation.md @@ -0,0 +1,81 @@ +# Step 4: Data Preparation (JSONL to Nested JSON) + +This step handles the transformation of the raw Shopify bulk result (JSONL) into a hierarchical JSON format that the OMS can process. + +## Control Flow (How we reach this step) +Control reaches this step immediately after the Shopify Bulk Operation is confirmed as `completed`. The sequence is as follows: + +1. **Confirmation (Step 3)**: The `process#BulkOperationResult` service identifies the operation is complete and calls `receive#IncomingSystemMessage`. +2. **Direct Trigger**: Since the `BulkProductAndVariantsByIdQuery` message type is configured with **`consume#ProductVariantUpdates`** as its consume service, the System Message framework triggers this step directly. +3. **Data Preparation**: This service (`consume#ProductVariantUpdates`) then reads the download URL, streams the JSONL, converts it to nested JSON, and uploads it to the MDM. + +## 1. Input: Raw JSONL Data +Shopify bulk results are returned as a "flat" JSONL file where child entities (Variants, Metafields) follow their parents and reference them via a `__parentId`. + +**Example (Simplified JSONL):** +```json +{"id":"gid://shopify/Product/9028462969133","handle":"doria-denim","title":"Doria - Denim"} +{"id":"gid://shopify/ProductVariant/47646836457773","sku":"5138980_denim_50","__parentId":"gid://shopify/Product/9028462969133"} +{"id":"gid://shopify/Metafield/31849382215981","key":"show_width_code","value":"G","__parentId":"gid://shopify/Product/9028462969133"} +``` + +## 2. Transformation Logic +The `consume#ProductVariantUpdates` service parses this file line-by-line and groups child entities under their respective parent objects based on the `__parentId`. + +- **Root Object**: Any line without a `__parentId` is treated as a new Product. +- **Child Object**: Any line with a `__parentId` is added to a list inside the current Product object. The list name is derived from the object type in the Shopify GID (e.g., `ProductVariant`, `Metafield`). + +## 3. Output: Nested JSON +The transformation produces a standard JSON array where each product contains its own nested variants and metadata. + +**Example (Transformed JSON):** +```json +[ + { + "virtualProduct": { + "id": "gid://shopify/Product/9028462969133", + "handle": "doria-denim", + "title": "Doria - Denim", + "ProductVariant": [ + { + "id": "gid://shopify/ProductVariant/47646836457773", + "sku": "5138980_denim_50" + } + ], + "Metafield": [ + { + "id": "gid://shopify/Metafield/31849382215981", + "key": "show_width_code", + "value": "G" + } + ] + } + } +] +``` + +## 4. Upload to MDM +Once the file is converted, it is uploaded to the **Data Manager (MDM)** for processing. + +- **Service**: `co.hotwax.util.UtilityServices.upload#DataManagerFile` +- **Config ID**: `SYNC_SHOPIFY_PRODUCT` +- **DataManager Configuration**: + ```json + { + "importServiceName": "co.hotwax.sob.product.ProductServices.sync#ShopifyProduct", + "executionModeId": "DMC_QUEUE", + "configId": "SYNC_SHOPIFY_PRODUCT" + } + ``` +- **Traceability**: The `systemMessageId` is passed in the `parameters` map to maintain a link between this data batch and the original sync request. + +## Service Call Chain +1. `co.hotwax.shopify.system.ShopifySystemMessageServices.process#BulkOperationResult` (Step 3: detects completion) +2. `org.moqui.impl.SystemMessageServices.receive#IncomingSystemMessage` (Creates incoming message and triggers consume) +3. `co.hotwax.shopify.system.ShopifySystemMessageServices.consume#ProductVariantUpdates` (This step: conversion & upload) +4. `co.hotwax.util.UtilityServices.upload#DataManagerFile` (Upload to MDM) + +## Technical Notes +- **Streaming**: The data is processed line-by-line using a `JsonGenerator` to handle potentially large files without exhausting memory. +- **Order Dependency**: The parser expects parents to precede their children in the Shopify file. +- **Traceability**: Uses `parentMessageId` (from previous steps) and passes `systemMessageId` to the Data Manager to ensure full end-to-end logging. diff --git a/project-ideas/product-master/new-design/step-5-diff-computation.md b/project-ideas/product-master/new-design/step-5-diff-computation.md new file mode 100644 index 00000000..e693eab8 --- /dev/null +++ b/project-ideas/product-master/new-design/step-5-diff-computation.md @@ -0,0 +1,60 @@ +# Step 5: Diff Computation (Change Detection) + +Once the nested JSON is received by the Data Manager, the core logic in `syncShopifyProduct.groovy` begins the process of identifying changes. Instead of blindly overwriting the database, it uses a **Baseline Comparison** strategy to perform selective updates. + +## 1. History Lookup (The Baseline) +For every Product or Variant in the incoming JSON, the service fetches the last synchronized state from the history table. + +- **Entity**: `co.hotwax.product.ProductUpdateHistory` +- **Lookup Key**: `productId` (Shopify GID) and `shopId`. +- **Purpose**: This record stores hashes and data snapshots from the *previous* sync, serving as the benchmark for change detection. + +## 2. Hashing Logic +To detect changes efficiently, the service groups product data into "buckets" and computes a **SHA-256 Hash** for each. + +| Data Bucket | Fields Included in Hash | +| :--- | :--- | +| **Core Details** | Title, Handle, Vendor, Featured Image URL, shipping/giftCard flags. | +| **Tags** | A sorted list of Shopify tags. | +| **Features** | A sorted list of product options (Position, Name, Value). | +| **Metafields** | A list of normalized metafields (Namespace, Key, Value). | +| **Assocs** | A sorted list of variant IDs associated with the product. | + +**Why Hashing?** Comparing a single 64-character hash is significantly faster than comparing dozens of individual fields or large JSON blobs. + +## 3. Delta Identification +If an incoming hash does not match the stored hash, the service performs a deep comparison to identify the exact delta. + +### List Comparison (Tags, Features, Metafields) +For lists, it identifies exactly which items are new and which have been removed: +- **`added`**: Items present in incoming data but missing in history. +- **`removed`**: Items present in history but missing in incoming data. + +### Field Comparison (Price, Title, etc.) +For simple fields like Price or weight, it performs a standard value comparison (e.g., using `BigDecimal` for price accuracy). + +## 4. Output: The `differenceMap` +The result of this step is a specialized JSON object called the **`differenceMap`**. This map is the "instruction set" for the next step (Ingestion). + +**Example `differenceMap`:** +```json +{ + "title": "Doria Denim - Updated", + "tags": { + "added": ["New Arrival", "Summer 2024"], + "removed": ["Sale"] + }, + "features": { + "added": [{"name": "Width", "value": "Wide"}], + "removed": [] + } +} +``` + +## Service Call Chain +1. `co.hotwax.sob.product.ProductServices.sync#ShopifyProduct` (Main Entry) +2. `compareVirtualProduct` / `compareVariantProduct` (Inner Logic) +3. `DigestUtils.sha256Hex` (Hashing Utility) + +## Traceability +The `systemMessageId` is carried forward and associated with the logic to ensure that every diff can be traced back to the specific Shopify sync request. diff --git a/project-ideas/product-master/new-design/step-6-database-updates.md b/project-ideas/product-master/new-design/step-6-database-updates.md new file mode 100644 index 00000000..7f2f10ac --- /dev/null +++ b/project-ideas/product-master/new-design/step-6-database-updates.md @@ -0,0 +1,39 @@ +# Step 6: Database Updates (Ingestion) + +In this step, the system takes the **`differenceMap`** generated in Step 5 and applies those changes to the actual Product entities in the database. This process is handled by a specialized worker logic within the same synchronization service. + +## 1. Selective Modification +The core principle of this step is **Selective Update**. The system only executes service calls for the specific fields or lists that are present in the `differenceMap`. This minimizes database load and prevents unnecessary triggers or audit log entries. + +## 2. Worker Logic (`consumeProductUpdateHistoryWorker`) +The logic is encapsulated in a worker closure that iterates through the changes. + +- **Entity Lookup**: It first resolves the internal `productId` from the Shopify ID using the `co.hotwax.shopify.ShopifyShopProduct` cross-reference table. +- **Handling New Products**: If no internal ID is found, it performs a full creation (mapping Shopify fields to the `Product` entity). +- **Applying Deltas**: It checks each key in the `differenceMap` and calls the corresponding Moqui service. + +## 3. Entity Mapping & Operations + +| Data Type | Internal Entity | Moqui Service / Operation | +| :--- | :--- | :--- | +| **Core Product** | `org.apache.ofbiz.product.product.Product` | `store#Product` (Updates Name, Image, Type, Internal Name). | +| **Tags** | `org.apache.ofbiz.product.product.ProductKeyword` | `create#ProductKeyword` for added tags; `delete#ProductKeyword` for removed tags. | +| **Features** | `org.apache.ofbiz.product.feature.ProductFeatureAppl` | Dynamically resolves/creates `ProductFeature` and creates/deletes standard applications. | +| **Pricing** | `org.apache.ofbiz.product.price.ProductPrice` | Updates the `LIST_PRICE` / `PURCHASE` record if the price changed. | +| **Identifications** | `org.apache.ofbiz.product.product.GoodIdentification` | Manages `SKU` and `UPCA` (Barcode) entries, including expiration (thruDate) of old values. | +| **Attributes** | `org.apache.ofbiz.product.product.ProductAttribute` | Maps non-standard metafields to product attributes. | + +## 4. Product Type Detection +The ingestion logic dynamically determines the `productTypeId` based on Shopify flags: +- `requiresShipping: false` -> **DIGITAL_GOOD** +- `hasVariantsThatRequireComponents: true` -> **MARKETING_PKG_PICK** +- Default -> **FINISHED_GOOD** + +## Service Call Chain +1. `consumeProductUpdateHistoryWorker` (The worker loop) +2. `store#org.apache.ofbiz.product.product.Product` +3. `store#org.apache.ofbiz.product.product.ProductKeyword` +4. `store#org.apache.ofbiz.product.feature.ProductFeatureAppl` + +## Traceability +Each database operation is performed in the context of the current sync session. Successful updates are verified before the system moves to the final state persistence step. diff --git a/project-ideas/product-master/new-design/step-7-history-persistence.md b/project-ideas/product-master/new-design/step-7-history-persistence.md new file mode 100644 index 00000000..3a7defe2 --- /dev/null +++ b/project-ideas/product-master/new-design/step-7-history-persistence.md @@ -0,0 +1,33 @@ +# Step 7: History Persistence + +The final stage of the product synchronization process is to update the history record with the latest state. This "closes the loop" and ensures that the next synchronization cycle has an accurate baseline for comparison. + +## 1. Updating the Baseline +After the database updates (Step 6) are successfully applied, the service persists the newly computed hashes and data blobs to the history table. + +- **Entity**: `co.hotwax.product.ProductUpdateHistory` +- **Action**: `store#co.hotwax.product.ProductUpdateHistory` + +### Fields Updated: +- **`productCoreDetailsHash`**: The new hash for basic product detail. +- **`tagsHash` / `featuresHash` / `metafieldsHash`**: Updated hashes based on the synchronized data. +- **`tags` / `features` / `metafields` / `identifications`**: The full JSON snapshots of the current state. +- **`differenceMap`**: The JSON delta calculated in Step 5. +- **`systemMessageId`**: The ID of the sync request that triggered this specific update. + +## 2. Importance of Persistence +Without this step, the next sync would re-detect the same changes, leading to redundant database updates and potential data flapping. By saving the history, we guarantee: +- **Idempotency**: Repeated syncs with the same data result in zero database changes. +- **Efficiency**: Only future changes on the Shopify side will trigger new work. + +## 3. Auditing & Troubleshooting +Storing the `differenceMap` and `systemMessageId` inside the history table turns it into a powerful audit trail. +- If a product value is unexpected in the OMS, developers can check the history table to see the exact `differenceMap` from the last sync and the corresponding `SystemMessage` that fetched the data from Shopify. + +## Sequence Order +1. **Ingest to Product Tables** (Step 6) +2. **Verify Success** (Implicit in script execution) +3. **Save to History Table** (Step 7) + +## Technical Note: Atomic Update +In the `syncShopifyProduct.groovy` script, the history persistence is the last operation before the service returns. This ensures that the baseline is only updated if the logic reaches the end of the script successfully. diff --git a/project-ideas/product-master/ShopifyNewProductsSyncDesign.md b/project-ideas/product-master/obsolete/ShopifyNewProductsSyncDesign.md similarity index 100% rename from project-ideas/product-master/ShopifyNewProductsSyncDesign.md rename to project-ideas/product-master/obsolete/ShopifyNewProductsSyncDesign.md diff --git a/project-ideas/product-master/ShopifyProductUpdatesSyncDesign.md b/project-ideas/product-master/obsolete/ShopifyProductUpdatesSyncDesign.md similarity index 100% rename from project-ideas/product-master/ShopifyProductUpdatesSyncDesign.md rename to project-ideas/product-master/obsolete/ShopifyProductUpdatesSyncDesign.md diff --git a/project-ideas/product-master/createProductAndVariants.md b/project-ideas/product-master/obsolete/createProductAndVariants.md similarity index 100% rename from project-ideas/product-master/createProductAndVariants.md rename to project-ideas/product-master/obsolete/createProductAndVariants.md diff --git a/project-ideas/product-master/createProductVariant.md b/project-ideas/product-master/obsolete/createProductVariant.md similarity index 100% rename from project-ideas/product-master/createProductVariant.md rename to project-ideas/product-master/obsolete/createProductVariant.md diff --git a/project-ideas/product-master/mapProduct.md b/project-ideas/product-master/obsolete/mapProduct.md similarity index 100% rename from project-ideas/product-master/mapProduct.md rename to project-ideas/product-master/obsolete/mapProduct.md diff --git a/project-ideas/product-master/mapProductVariant.md b/project-ideas/product-master/obsolete/mapProductVariant.md similarity index 100% rename from project-ideas/product-master/mapProductVariant.md rename to project-ideas/product-master/obsolete/mapProductVariant.md diff --git a/project-ideas/product-master/prepareProductCreate.md b/project-ideas/product-master/obsolete/prepareProductCreate.md similarity index 100% rename from project-ideas/product-master/prepareProductCreate.md rename to project-ideas/product-master/obsolete/prepareProductCreate.md diff --git a/project-ideas/product-master/prepareProductUpdate.md b/project-ideas/product-master/obsolete/prepareProductUpdate.md similarity index 100% rename from project-ideas/product-master/prepareProductUpdate.md rename to project-ideas/product-master/obsolete/prepareProductUpdate.md diff --git a/project-ideas/product-master/updateProductAndVariants.md b/project-ideas/product-master/obsolete/updateProductAndVariants.md similarity index 100% rename from project-ideas/product-master/updateProductAndVariants.md rename to project-ideas/product-master/obsolete/updateProductAndVariants.md diff --git a/project-ideas/product-master/updateProductVariant.md b/project-ideas/product-master/obsolete/updateProductVariant.md similarity index 100% rename from project-ideas/product-master/updateProductVariant.md rename to project-ideas/product-master/obsolete/updateProductVariant.md